Skip to content

Recent Articles

18
Apr

Doing iframe resize

As it happens, I had to integrate some external applications through iframes, and I re-discovered how much this sucked.

There is a lot of scripts out there, but I couldn’t find one that works for me, so basically I had to write my own. Let me introduce you wrapframe, which aims at a seamless integration of an iframe into a document. It’s only an aim though, I don’t think it is ever going to be possible to do it right.

You can get the project scope and many other things on the project’s README, however there is a few details I’d like to discuss here.

Watching the size of an element

In many cases you might have to watch the size of something, in order to react in case its size changes. Like change the zoom on a map depending on its own size, or in the present case get the size of an iframe’s content.

As all developers have been taught, polling is bad and should be avoided at all costs. Events are much better. Only here, there is just so many events, you can’t really track them down. The window can be resized, a dom element might be added/removed, a style might change, and so on. The possible changes are endless.

So what I advise is the simplest and the most fool-proof solution: just fucking poll. Watch the size of your thing every 1/60th of second, and if you notice any change just trigger your stuff.

Another variant which can be relevant in some cases (like when it is heavy to do a resize), you can wait for about 200ms that the size stabilizes itself, and then trigger your resize actions. This helps not killing the CPU when the user resizes the window, by example.

Get the size and position of a DOM node

There is a lot of things you see, with the scrollHeight, offsetHeight and so on. Although those have their use, they are clearly messy and deceptive. Moreover, you won’t really get the position of your element with them.

Of course, you could use jQuery, but it’s not always available to save you. Or maybe you just find it too slow.

A solution exists: there is the getBoundingClientRect() method. It exists on all browsers, behaves consistently and brings no surprises along. And it’s pretty fast.

Get the size of a page

It’s a fucking mess. Most if it is detailed in the project’s readme. I’ll just drop a few words here.

Basically, you have to know pretty well how your page works and what you’re going to do with it. Then you will know which elements are representative of your page height, and then you will know what to measure.

Cross-domain communcation

Not much to be said. The top solutions out there are

  • Hash-based
  • Flash-based
  • postMessage() based
  • hybrid

And basically, postMessage() is available in all current browsers, so it’s the only viable solution. No need to use a bloated hybrid approach, nor infamous hacks on the URL hash.

Conclusion

I hate Internet Explorer. Really, from the bottom of my heart. I hate it.

Appart from that, there is a lot of tricks floating around, but it has been a real pain to sort them out and find which one were usefuls. To sum-up

  • Cross-domain communication is neatly done with postMessage()
  • Getting the size and position of an element is really easy using getBoundingClientRect()
  • There is no way to see if an element’s size changed appart from polling it
  • The method to get a page’s height depends on its content and flow

Finding all that and trying all the other solutions have taken me quite a bit of time, don’t make the same mistakes as I did :)

15
Mar

Sorting JSON fields in PostgreSQL

PostgreSQL in its version 9.3 comes with a new data type, the JSON fields. If I had to give my opinion on it, to stay objective and moderate I’d say it is fucking awesome. A lot of other blog posts and documentation exist on the matter, so I won’t be covering this too much, but let’s just say that it enables you to make queries deep inside the JSON, like

SELECT * FROM mytable WHERE myjsonfield->>'title' = 'Look For This Title';

Now the only issue with that is that if like me you use Django, it is not yet fully compatible with PostgreSQL advanced features. Indeed, since it tries to provide a uniform set of features across all DB engines, its features are bound to what MySQL can do, aka barely key/value storage. Although it is in my opinion a huge mistake, there is currently an effort to properly support PostgreSQL in Django, and this is an awesome news.

However, in the meantime we’re back to using hacks, and one of these hacks is to use django-jsonfields, which unfortunately isn’t bullet proof. And in particular, if you happen to do a .distinct() on a queryset that selects a JSON field, you will be nicely warned by PostgreSQL that you can’t fucking compare two JSON fields, and thus it is impossible to know which values are distinct. Actually, I don’t even know why the ORM is bothering to do it this way, since all that needs to be unique is the ID, but nevermind. Changing the way the ORM works is just a pain in the arse, and I really wanted to avoid it.

And there comes this StackOverflow thread. It shows how to add the support for a comparison operator to the JSON field, which is just great. However, the given implementation suffers several issues

  • An expression like SELECT '{"a":1,"b":2}'::json = '{"b":2,"a":1}'::json returns false, because it is based on the string representation. This was a great idea, but unfortunately there is no way to make JSON.stringify() behave in a deterministic manner
  • It does not allow for comparison, but only for hashing. If the field stands alone this is not an issue, but if you start to mix it with other fields for your .distinct(), PostgreSQL will start to want it to be sortable. And since it isn’t, you will get a nice fail message.

So, the concept was good, but the implementation had to be different. Then I decided to create a json_cmp() function that would be able to compare (in the sense lower/greater than) two JSON objects, and that would power all the operators required for a b-tree operator class.

Because JSON is Javascript, I decided to code the json_cmp() function in Javascript, which requires to activate the PL/V8 extension.

CREATE EXTENSION plv8;

Then, you need to create the following bunch of objects

CREATE OR REPLACE FUNCTION json_cmp(left json, right json)
RETURNS integer AS $$
    function cleverType(obj) {
        var type = typeof obj;

        if (type === 'object') {
            if (obj === null) {
                type = 'null';
            } else if (obj instanceof Array) {
                type = 'array';
            }
        }

        return type;
    }

    function cmp(left, right) {
        var leftType = cleverType(left),
            rightType = cleverType(right),
            i,
            buf,
            leftKeys,
            rightKeys,
            output = 0;

        if (leftType !== rightType) {
            output = leftType.localeCompare(rightType);
        } else if (leftType === 'number'
                || leftType === 'boolean'
                || leftType === 'string') {
            if (left < right) {
                output = -1;
            } else if (left > right) {
                output = 1;
            } else {
                output = 0;
            }
        } else if (leftType === 'array') {
            if (left.length !== right.length) {
                output = cmp(left.length, right.length);
            } else {
                for (i = 0; i < left.length; i += 1) {
                    buf = cmp(left[i], right[i]);

                    if (buf !== 0) {
                        output = buf;
                        break;
                    }
                }
            }
        } else if (leftType === 'object') {
            leftKeys = Object.keys(left);
            rightKeys = Object.keys(right);

            if (leftKeys.length !== rightKeys.length) {
                leftKeys.sort();
                rightKeys.sort();
                buf = cmp(leftKeys, rightKeys);
            } else {
                buf = cmp(leftKeys.length, rightKeys.length);
            }

            if (buf !== 0) {
                output = buf;
            } else {
                for (i = 0; i < leftKeys.length; i += 1) {
                    buf = cmp(left[leftKeys[i]], right[leftKeys[i]]);

                    if (buf !== 0) {
                        output = buf;
                        break;
                    }
                }
            }
        }

        return output;
    }

    return cmp(left, right);
$$ LANGUAGE plv8 IMMUTABLE STRICT;

CREATE OR REPLACE FUNCTION json_eq(json, json)
RETURNS BOOLEAN LANGUAGE SQL STRICT IMMUTABLE AS $$
    SELECT json_cmp($1, $2) = 0;
$$;

CREATE OR REPLACE FUNCTION json_lt(json, json)
RETURNS BOOLEAN LANGUAGE SQL STRICT IMMUTABLE AS $$
    SELECT json_cmp($1, $2) < 0;
$$;

CREATE OR REPLACE FUNCTION json_lte(json, json)
RETURNS BOOLEAN LANGUAGE SQL STRICT IMMUTABLE AS $$
    SELECT json_cmp($1, $2) <= 0;
$$;

CREATE OR REPLACE FUNCTION json_gt(json, json)
RETURNS BOOLEAN LANGUAGE SQL STRICT IMMUTABLE AS $$
    SELECT json_cmp($1, $2) > 0;
$$;

CREATE OR REPLACE FUNCTION json_gte(json, json)
RETURNS BOOLEAN LANGUAGE SQL STRICT IMMUTABLE AS $$
    SELECT json_cmp($1, $2) >= 0;
$$;

CREATE OPERATOR =  (LEFTARG = json, RIGHTARG = json, PROCEDURE = json_eq);
CREATE OPERATOR <  (LEFTARG = json, RIGHTARG = json, PROCEDURE = json_lt);
CREATE OPERATOR <= (LEFTARG = json, RIGHTARG = json, PROCEDURE = json_lte);
CREATE OPERATOR >  (LEFTARG = json, RIGHTARG = json, PROCEDURE = json_gt);
CREATE OPERATOR >= (LEFTARG = json, RIGHTARG = json, PROCEDURE = json_gte);

CREATE OPERATOR CLASS json_ops
   DEFAULT FOR TYPE json USING btree AS
   OPERATOR 1 <,
   OPERATOR 2 <=,
   OPERATOR 3 =,
   OPERATOR 4 >=,
   OPERATOR 5 >,
   FUNCTION 1 json_cmp(json, json);

As you can see, the json_cmp() function tries to make somewhat meaningful comparisons, but of course this can’t always be the case. What matters is that it is totally deterministic and won’t return 0 unless the compared objects are strictly equals. Please note that the underlying JS function is tailored for JSON and V8, it will probably not be suitable for a general-purpose deep object comparison.

Also, be warned that the comparison seems to be somewhat slow if your JSON objects are big and nested, but it actually seems quite logical.

After creating those functions and operators, the following concept snippets should work:

-- Compare JSON fields (returns true)
SELECT '1'::json < '2'::json;

-- Sort a table according to a JSON field
SELECT * FROM mytable ORDER BY myjsonfield;

Please note that if you are a Django South user, you can create an empty migration for your app (./manager.py schemamigration myapp install_json_comparison --empty), and just replace the body of the forwards() method by something in the mood:

    def forwards(self, orm):
        db.execute("""
            -- All the SQL shown above
        """)

And voilà, with this Django’s ORM should be able to use your table as it pleases him!

Anyway, I am really amazed by PostgreSQL, which proves to be really easy to extend. In order to remotely approach this result using MySQL, you would need to spend days writing a C extension, whereas there you can just stack up your little piece of Javascript on top of already awesome components, and this makes the magic. I think I’m in love <3

6
Mar

Gibi ― A random word generator

I have recently been generating a fake world map, with fake cities and so on. As this was dull and boring, I searched something to improve my day, and finally remembered this Daily WTF post. It might not have been a great success for the other guy, but anyway it’s still funny to do.

So I wrote a few Python lines, got the list of french cities and started the fun.

A Gibi from « Les Shadoks » :)

A Gibi from « Les Shadoks » :)

After a bit experimenting and turning around, I came up with the gibi python package. The installation is pretty straightforward

pip install gibi

Then all you have to do is to use the gibi command to either generate your markov chain matrix, either generate random words. It turns out that a few things do improve the result

  • Filter out the words too short or too long (within [3, 30] character long seems nice)
  • Do not only consider the last character when generating the output, but the last 3 characters. The number can vary, but I find that the results with 3 are the best.

You can also use it as an API, as shown on Github.

Another bonus feature: you can seed the PRNG, meaning that you can produce a deterministic result given an input. Which helps me to generate always the same city names across my runs. It could also become some way of hashing stuff, especially in use with a cryptographically stronger generator like Skein, although I seriously doubt the security strength as well as the use for this technique.

A few words, for the end:

% gibi generate -n 15 french_cities.gibi
fontois
che
pouchazoud
malvadoux
chac
vieilhac
mesle
mirat
gardon
garronquil
inançon
orvées
ganda
ville
bun

This all sounds pretty useless, but anyway it was fun!

14
Jan

Thumbor and Heroku

Since I am currently working on Good Morning Planet, and that we will handle a lot of user pictures, that will have to be resized, came the question of how to generate thumbnails.

As always, probably due to my inner nature, I prefer lazy solutions. In this case, what I will be using is Thumbor, which is an awesomely incredible thumbnails generator. It will detect faces and features in pictures before cropping, and will generate stuff on the fly.

I’ll let you see the details, but in short it’s exactly what I intended to code from scratch, only better.

Now I want to use this on Heroku. But the smart cropping feature requires the OpenCV Python module, which is not bundled in the default Heroku stack, and not available through pip, because it’s a real hell to compile.

So, what I did instead was to create a custom buildpack: Jetpack. It is designed to be modular, though essentially to build my Python/Django apps.

On top of this, I have created a dedicated “prepack” (see Jetpack’s readme) and put up a custom configuration (also explained in its own readme).

Overall, the Jetpack provides a way to run Python + OpenCV, and the Thumbor Heroku configuration provides inspiration if you ever want to deploy Thumbor on Heroku. Please note that this has never been tested in production, even though this will be done soon!

Happy hacking :)

5
Oct

URL regular expression

Well, yes it’s old as the world, but whatever it’s still being useful.

My issue here is to match an URL from a text, in order to auto-replace it with a link (say in a messaging application). I try by no mean to validate it!

What you’re getting is the following: (^|\s)((f|ht)tps?://([^ \t\r\n]*[^ \t\r\n\)*_,\.]))

It will basically everything that is not a white character after http://, but also tries to detect things like URLs between parenthesis and other punctuation. So if you write something like Check out http://google.com/, it's awesome!, it will not include the , in the matched URL.

By the way, in order to do that I used Debuggex, which is getting really handy to work with regular expressions!

That’s it for today :)

26
Jun

IE6 invented the Web 2.0

I just wanted to share a small revelation I had, and it all starts with the XMLHttpRequest. First of all, the naming of it is quite weird: it’s XML but Http. Who might be so inconsistent to name an object this way? It should rather be XMLHTTPRequest or XmlHttpRequest… I can sense bullshit here, might Microsoft be behind that?

Well, it turns out that Microsoft did indeed introduce this object, and it was named XMLHTTP. And actually the wrong name comes from Mozilla, but Microsoft still invented it. Damn. (Wikipedia for further information)

But wait a second and think about the implications of this. It’s true that there is many things in browsers, more and more appearing every day. There is a whole galaxy of HTML5, CSS3 and JS APIs out there just waiting to be discovered by developers to do awesome things. But the cornerstone of this all, the one behind the whole AJAX hype that went on 10 years ago, is purely and simply our XMLHttpRequest friend!

And there is more. Indeed, did you notice the hell that it is to do a HTML5 website for IE6? All those polyfills and compatibility libraries in order to emulate HTML5 features, that’s so much a pain in the ass right? But you know that IE6 was released in 2001, do you think you could even come close to this with Phoenix 0.1 that was released in 2002? That’s right, Microsoft managed to make a browser so extensible that it lasted 10 years.

Then in case you were wondering, yes IE6 did pretty much invent the Web 2.0.

22
May

Frenetic Bunny

So, I’m currently doing something that I love (find the sarcasm if you can): a Facebook application. And as always, the real mess comes from the permissions system. Indeed, you don’t simply log your user in and then roll. Nope. The user has to grant you permissions over its private data, starting from its own identity to his marital status.

In order to do that, you have to load the Facebook API, initialize it, and then somehow once you know that the FB object is there you can start using it and try to use the FB.login() method, although the user might not grant you all the permissions you asked for and it’s going to be a pain in the ass. So, enough is enough, and screw that shit.

I decided to use the principle of “promises” that you can find in jQuery, and to code my whole library around that. So, the key of it is that you ask the library to promise you that you’ll get the user’s drunk pictures, and that will asynchronously warn you whenever the user accepted that or asked you to go to hell. Incidentally, it also allows you to run some code exclusively when Facebook is properly loaded, so you can be sure to have the FB object available.

So, in short. Once you’ve been through a bit of init code (you really can’t skip it), what you have to write is simply:

$.fb.connected('email').done(function () {
    // Do something useful here now that your user is connected
}).fail(function () {
    console.log('The user did reject us!');
});

This will ensure that you have the appropriate permission, or if you don’t, it will prompt the user to give it to you. Nothing revolutionary, but that’s still a lot nicer to use than the raw API.

Please note that you can also do that:

$.fb.ready(function () {
    FB.someShit();
});

This makes sure that FB is correctly loaded and initalized before you call the API.

For more details, and the code of the library itself, you can head to the GitHub project!

24
Mar

Slumdog Maiznet

This week-end I participated to the Federez days as a Maiznet Representative, and it was quite an interesting exchange of ideas and experience with other student network associations. If you are a French student network association, you should be joining Federez :)

I also happen to have been giving a presentation about what I learned and remembered in Maiznet over my years of participating and running it. You will find the slides over here :

I’m sorry for non-French speakers, but this time it’s plain French all over ;)

Now Maiznet and all that stuff is quite behind me, however I’m glad I had a shot at doing this!

30
Jan

Python run-time mixins

Don’t ask my why, I started considering to use some sort of “dynamic mixins”: what if you want to be able to mix a class into another during run-time, only for a single instance of my object. To illustrate this, here is some code example of what I try to do:

class Blendable(Blender):
    pass

class BlendMe(object):
    def works(self):
        print("Works for a")

a = Blendable()
b = Blendable()

a.mix_in(BlendMe)

# a should have the "works" method
a.works()

# but b shouldn't
try:
    b.works()
except AttributeError:
    print("Works for b")

Well, I have implemented this Blender class, here’s the result:

class Blender(object):
    def mix_in(self, chili):
        if not isinstance(chili, type):
            raise TypeError(
                "Blender.mix_in(): you can only mix in types, got %s.%s instead" % (
                    chili.__class__.__module__,
                    chili.__class__.__name__
                )
            )

        newtype = type(
            "%s_%d" % (self.__class__.__name__, id(chili)),
            self.__class__.__bases__,
            dict(self.__class__.__dict__)
        )

        newtype.__bases__ = self.__class__.__bases__ + (chili,)
        self.__class__ = newtype

I’m not sure I’m going to use that for real, but the concept is surely funny. Leave me some comment if you ever need to use something as crazy as this :)

You can get the full file here.

12
Dec

HTML5′s localStorage as software bus

I keep wondering what goes through my mind. Today I stumbled across HTML5′s localStorage facility, and I was immediately seduced by the possibilities it offers. I don’t know which ones, but surely they are huge!

One of the things that I could think of and that might open up a whole new sky is that thanks to the “storage” event, you can actually use this localStorage as a software bus between your Javascript pages. IPC in the browser! Am I too enthusiastic? Probably, but I’ll still show you a small example that I put up and that I think is quite fun :

All right, you think it is a dead text field? Just try opening this page again simultaneously in another window. Put the two text boxes side to side, and start typing into one of them. Profit :)

How does the magic work? Easy, you can store any string into the localStorage object (just use it as an array), and when it is modified by one page, all the other pages receive a storage event that tells them that a new value was added. So basically, the code to do this is extremely simple. Here are the useful parts of it, made out of jQuery and Angular.js for the sake of awesomeness.

                <div class="container-fluid">
                        <div class="main-box" ng-controller="localBusCtrl">
                                <h2>This field is...</h2>

                                <input type="text" ng-model="busValue">
                        </div>
                </div>
(function ($, ng, localStorage) {
	'use strict';

	ng.module('localBus', [])
		.controller('localBusCtrl', function ($scope, $window) {
			$scope.busValue = localStorage.busValue || "magic";

			$scope.$watch('busValue', function () {
				localStorage.busValue = $scope.busValue;
			});

			$($window).on('storage', function (e) {
				$scope.$apply(function () {
					if (e.originalEvent.key === "busValue") {
						$scope.busValue = e.originalEvent.newValue;
					}
				});
			});
		});
}(jQuery, angular, localStorage));

Basically, how does it work?

  • Read the initial value from the localStorage (which is persistent across page reloads etc)
  • If the input field gets modified, write the new value into localStorage
  • If you get a storage event, update the input field’s value

Couldn’t be any simpler :)

Now that this is done, I wonder what else could be invented. Mutualize XHR between instances of a page? Synchronize some content? Create a chat system for schizophrenic users? Time will tell if I find anything useful to do from this!

Many thanks to Dive Into HTML5, from which I got everything I needed to know about the localStorage!