We recently released a very slick Firefox extension at Wesabe. It was written by my colleague Tim Mason, but I helped figure out one small piece of it—namely, how to do binary multipart POSTs in Javascript—and since it involved many hours of hair-pulling for both of us, I thought I’d share the knowledge. (Tim says I should note that “this probably only works in Firefox version 2.0 and greater and that it uses Mozilla specific calls that are only allowed for privileged javascript—basically only for extensions.”)
One of the cool features of the plugin is the ability to take a snapshot of a full browser page and either save the snapshot to disk or upload it to Wesabe (so you can, for example, save the receipt for a web purchase along with that transaction in your account). The snapshot is uploaded to Wesabe via a standard multipart POST, the same way that a file is uploaded via a web form.
Tim was having trouble getting the POST to work with binary data at first, and he had other things to finish, so he wanted to just base-64-encode it and be done with it. I was reluctant to do that, as the size of the upload would be significantly larger (about 137% of the original). Also, Rails didn’t automatically decode base-64-encoded file attachments. But Tim had other bugs to fix, so I submitted a patch to Rails to do the base-64 decoding. I was pretty proud of this patch until it was pointed out to me that RFC 2616 specifically disallows the use of Content-Transfer-Encoding in HTTP. Doh. They also realized that it is a colossal waste of bandwidth.
Since Tim was cramming to meet a hard(-ish) deadline set for the release of the plugin, I offered to lend my eyeballs to the binary post problem. This could be a very long story, but I’ll just get to the point: you can read binary data in to a Javascript string and dump it right out to a file just fine, but if you try to do any concatenation with that string, Javascript ends up munging it mercilessly. I’m not sure whether it is trying to interpret it as UTF8 or if it terminates it as soon as it hits a null byte (which is what seemed to be happening), but regardless, doing "some string" + binaryData + "another string", as is necessary when putting together a mutipart post, just does not work.
The answer required employing Rube Goldbergian system of input and output streams. The seed of the solution was found on this post, although that didn’t explain how to mix in all of the strings needed for the post and MIME envelope. So here it is, in all it’s goriness:
Update: Spaces removed from multipart boundary per Gijsbert’s suggestion in the comments (thanks!).
I ran across a bug in Date::Format today, and after spending a few hours hacking away at a fix (the date/format.rb code is uuuuugly and sloooow…someone should really rewrite that. Better yet, rewrite it in C), I thought I’d submit a patch. So I grabbed the ruby_1_8 branch and lo and behold, my issue had already been fixed!
So the question now was how to monkeypatch the entire Date::Format module? Simply require-ing it as a plugin doesn’t work, since Date::Format is already loaded at that point. The trick then is to use load instead of require.
First, I tried this:
load'date/format.rb'
(note: load needs the actual filename; it doesn’t have the magic that require does) but that gave me an error:
in `load': wrong number of arguments (1 for 0) (ArgumentError)
Turns out Rails::Plugin::Loader defines its own load, so:
It took me a bit of digging to find this, so I thought I may as well spread the word here. I’ve started using Adobe Lightroom for managing my photos, but every time I connect my camera or card reader to import photos, iPhoto would launch, and I couldn’t figure out where I could change that. I finally discovered that the setting is in the Image Capture application. Run that, then go to Preferences and you’ll find it there. Why they wouldn’t also put it in iPhoto, Lightroom, or even on a preferences panel is beyond me.
Marc and I presented our Super Ninja Privacy Techniques for Web App Developers talk again yesterday at Web 2.0 (we also did it at ETech last month), and it seemed well received. I still need to work on my presentation skills; I’m not an old hand at this like Marc is.
I’m working on a follow-up to my privacy wall post which will describe a much better way to go about keeping a user’s private data private, using an encrypted “Locker”. I’ll also go into detail about how we deal with password recovery.
Hopefully by preannouncing this, I’ll force myself to get off my duff and finish the post. So stay tuned, and bug me if I don’t have it up soon.
Google recently announced that it will soon start anonymizing search logs older than 18-24 months. Full details can be found in their Log Retention Policy FAQ (PDF). This is a heartening step back towards their “Don’t Be Evil” corporate philosophy, which some think has been largely abandoned.
I’ve just recently started using Scroogle as a way of defeating their tracking of my every search (their site is awful; Wikipedia has more readable information about the project), although the motives of the man behind it, Daniel Brandt, who also runs the Google Watch site, may be questionable. Still, he doesn’t have much incentive for keeping a log of queries and IP addresses, and if he did, since he’s not giving me a cookie, he can’t tie all my searches together.