My friends Sunah Suh, Anthony Hersey, and I finally figured out the magic incantations needed to automate sending “STOP” to political text messages. Setting it up is completely non-obvious, so I’ll walk through it step-by-step. [read]
One of the engineers on my team at Etsy, who is working on getting our clickstream events into protobuf, noticed that we were getting some values that were too large for protobuf’s largest integer, uint64. E.g: 55340232221128660000. [read]
Advent of Code was a fun diversion this year. It seemed a bit easier than some previous years. Perhaps it helped that I picked a language that I had some experience with, Swift. Here are some quick thoughts on Swift as an AoC language, some general AoC tips, and notes on some of the more interesting puzzles. [read]
I scratched an itch last week and actually wrote some code (these days my primary IDE is Google Docs). Check out humanize, which replaces any large numbers in text piped into it with shortened versions using SI prefix symbols (K, M, G, T, etc.). So “File size: 972917401236” becomes “File size: 973G”. [read]
I wrote a Slack bot, ReadBot, to both scratch an itch and check out the current state of Slack integration. As a bonus, I got to spend some time getting familiar with Glitch, a pretty cool live coding platform. [read]
I recently got a new monitor and it came with a problem. For some reason when I attached or detached my laptop, many of my windows would end up off the edge of the screen, and I’d have to drag them back over one by one. It was annoying enough that I decided I’d write an app to “rescue” them. [read]
When my friend Nelson and I set out to build our location tracking app & accompanying website, Wanderings, we started down the familiar path of choosing our stack. Flask seemed like a reasonable choice. Postgres for the database. SqlAlchemy for the ORM? Sure, I guess. We figured we’d try hosting on GCP, mostly to learn more about the platform. Do we bother with containers? Meh, leave those for the youngsters. And so on. [read]
This is one of those I-spent-long-enough-stumped-on-this-issue-I-should-write-it-up-for-future-generations posts. [read]
I wrote a Mac menu bar app for internal use at Etsy that show the status of a number of internal systems and pops up a notification if any of them change status—for example, if a problem is detected and someone puts a hold on our deployment queue. [read]
Etsy’s culture puts a very strong emphasis on work-life balance. As someone who has mostly worked for startups, this took some adjusting to. The fact that Etsy IRC and email pretty much goes silent outside of normal business hours struck me as odd initially. But, I really appreciate it. Startups are stressful, and they left me feeling that there was always something work-related that I should be doing whenever I had some “free time.” [read]
I ran across two apps yesterday that make packaging Mac apps for distribution a breeze: [read]
This past week I taught an Etsy School class on how to write an OS X menu bar (aka status bar) app in Swift, and the walk-through is now public. I hope you find it useful. [read]
I’m a big fan of the Sublime Text text editor, and I’ve become one of its biggest evangelists at Etsy. A few months ago I gave an internal presentation/tutorial session on it, and used those notes to create a guide on our internal wiki. I’ve ported that guide to my blog so I can spread the Good Word to others. Since the source code for my blog is available publicly, I’m also hoping that people will help make it even better. [read]
Jekyll’s Latent Semantic Indexing (LSI) indexes your posts to create the “Related Posts” section you see at the bottom of the individual post pages. However, if you have more than a handful of posts, it is very slow, and it seems to get exponentially slower as the number of posts increase. I ran it on this site, with ~140 posts, and I finally killed it after 3 hours. [read]
Swift is a great language, and far more enjoyable to program in than Objective-C, but it is still in its infancy, and bugs abound. One of the more frustrating is compiler errors that are flat-out wrong. Take a look at this (in Xcode 6.1): [read]
One of the defining charactaristics of a successful programmer is being OK with the feeling that you have no idea what you’re doing. I’ve been programming for over 30 years and it still happens to me regularly. The key is pushing on through that dark period, trusting that eventually you’ll catch a glimmer of light and it will slowly start to make sense. [read]
Prose.io is a content editor for GitHub. It supports Jekyll sites (like this one), and provides Markdown previews. I’m writing this post using it. [read]
The most dangerous thought that you can have as a creative person is to think that you know what you’re doing. Because once you think you know what you’re doing, you stop looking around for other ways of doing things. [read]
Every Git user, from novice to expert, gets this error message at some point, maybe even daily: [read]
Excellent explanations of some complex math & programming topics. My favorite: An Interactive Guide To The Fourier Transform. [read]
Amazing multimedia (that word is so 90s, but nothing more appropriate comes to mind) story of a fatal avalanche at Stevens Pass. Just look. [read]
I don’t know how I managed to avoid learning about John H. Hammond until now. A recent Radiolab episode, Crossroads, introduced him to me. [read]
“Research strongly suggests that people are more creative when they enjoy privacy and freedom from interruption.” [read]
I started working with python full-time about a year-and-a-half ago. I was coming from having mostly done ruby and javascript for the last few years, and my prior exposure to python was relatively brief. [read]
“To be the father of growing daughters is to understand something of what Yeats evokes with his imperishable phrase ‘terrible beauty.’ Nothing can make one so happily exhilarated or so frightened: it’s a solid lesson in the limitations of self to realize that your heart is running around inside someone else’s body. It also makes me quite astonishingly calm at the thought of death: I know whom I would die to protect and I also understand that nobody but a lugubrious serf can possibly wish for a father who never goes away.” [read]
A recent article by Mikeal Rogers about the Apache Software Foundation’s outmoded idea of open source contributions struck a chord with me. [read]
Interesting post from Andy Baio about tracking down an anonymous blogger via Google Analytics, and the trickiness of remaining anonymous in general. [read]
Interesting approach, showcasing the power of open source machine learning and computer vision libraries. [read]
Someday I’ll write my own take on parenthood, and I hope it’s half as good as Jeff Atwood’s. [read]
Proof-of-concept out of Georgia Tech using the iPhone 4’s accelerometer as a keylogger. [read]
This is a special collection of problems that were given to select applicants during oral entrance exams to the math department of Moscow State University. These problems were designed to prevent Jews and other undesirables from getting a passing grade. Among problems that were used by the department to blackball unwanted candidate students, these problems are distinguished by having a simple solution that is difficult to find. Using problems with a simple solution protected the administration from extra complaints and appeals. This collection therefore has mathematical as well as historical value. [read]
Dart is a new web programming language from Google. See also the blog post introducing Dart. [read]
“Market research is what you do when your product isn’t any good.” [read]
My first home computer was an Apple IIe, in 1984. I had already started learning to program (on a TRS-80 Model III), but that IIe was where I really got excited about computers and programming. I wrote utilities and games in BASIC, then 6502 assembly language (I would write the assembler instructions down on paper, then translate those by hand to the byte code), hacked games (Bard’s Tale character editor, anyone?), and even taught programming classes to other kids. [read]
See also OWASP’s Cheat Sheet Series. [read]
Mentioned previously, O’Reilly’s Open Feedback Publishing System is a great resource for developers, and a brilliant move my O’Reilly to get early feedback and corrections on their books before they go to print. [read]
Qu is a new Ruby queuing and background job-processing library from Brandon Keepers, one of the maintainers of delayed_job. [read]
Fascinating article about a New York Times writer who moves to Russia with his family, and immerses his kids in a Russian school. [read]
A great collection of “off the grid-for-little-kids” videos. “It’s most likely not stuff that was made for them…But we don’t underestimate kids around here.” [read]
Great guide to HBase, available free (for the time being) online via O’Reilly’s Open Feedback Publishing System. [read]
Pandoc is a universal document converter. Incredibly useful, especially for converting Markdown to HTML, PDF, etc. [read]
My Yahoo account was hacked this morning. The first sign that something was up was an email from Yahoo saying that an email address was added to my account. The address was “bradley_greenlee@yahoo.com”. Tricky. It definitely gave me pause. That’s my name, but I didn’t think I had that account. I went in and deleted it. Suddenly Adium went nuts, popping up 20+ dialogs saying “<username> has (retroactively) denied your request to add them to your list.” Huh. Then came the first of a number of emails from friends (basically, Y! IM friends) saying that they had gotten an email from me saying that I was stuck in Wales without my creditcard and passport yada yada. Yup, hacked. I was already on my way to changing my password. Unfortunately, my Yahoo account was still tied to my old AT&T DSL account, and I had to remember the username for that before I got in (this actually probably saved my butt, preventing the attacker from changing my password). I did eventually get in, though, and changed it to a nice, long random string. So, how did this happen? Well, Yahoo was my spam-bucket account–I used it on a lot of throwaway sites that needed an email address. Unfortunately, I never changed the password from a common one I use on a lot of those same throwaway sites (for sites that I actually care about, I generate a random password, which I store in 1Password). Stupid. I’m sure one of those sites got compromised and were storing their passwords in plaintext. I don’t think much damage was done, other than spamming a lot of Y! IM friends. The upside was that I got back in touch with some friends I hadn’t talked to in ages. The lesson here is to never trust any site to keep your password secure. Count on it getting compromised at some point. Use a random password for each one, and a tool like 1Password or LastPass to keep track of them. Most important, never use the same password on your email accounts that you do on any other site. If your email account gets compromised, the attacker can use that account to gain access (via “forgot password”) to any other sites that use that email address. [read]
ReadWriteWeb posted a short article today, The iPad Not a Savior of Magazines, as Digital Sales Continue to Fall. Looking at the current sales trend is pretty silly, however. Of course you’re going to see a big drop-off. Early adopters are eager to check out The Future, but people aren’t going to continue to pay newsstand prices for digital editions of magazines, especially if they’re already paying for a print subscription. I don’t think we’ll see much improvement until both Apple and the publishers get over themselves and come to an agreement on subscription pricing. At issue is how much access the publishers get to their subscribers’ data. Traditionally, what keeps print publications alive is their ability to resell their subscribers’ information to advertisers. But Apple isn’t about to give that up. So either publishers need to accept that they’re going to have to give up some control in order to play in Apple’s ecosystem, or Apple needs to loosen its grip a bit (less likely). Still, I’d wager that in 5 years, digital subscription sales will surpass print. [read]
http://www.avclub.com/articles/neutral-milk-hotel,38877/The Swell Season covers Neutral Milk Hotel [read]
It took me a bit of digging to figure this out (it kept failing with readline errors, and then iconv went missing), so I thought I’d share: [read]
For Jessica: A couple of weeks ago, a friend of mine and I were talking about a study she'd just read, which concluded that people without children were happier than people with children; or, to put it more precisely, despite what conventional wis... [read]
I recently set up a Mac Mini at home, and have found using Screen Sharing to access it invaluable. Not wanting to open up anything other than SSH on my home router, I created the following script to open up a Screen Sharing session over SSH: [read]
Peter Callesen - A Single Sheet of Paper [read]
Richard Feynman and The Connection Machine: I heart Feynman [read]
Christoph Niemann's Abstract City [read]
For some time now, Wesabe’s transaction edit dialog has been broken in Opera (10.10 as of this writing). Since it was fine in all other major browsers, and none of us were Opera users, fixing it was never high priority. I thought I’d take a look at it again tonight, though, and just when I was about to give up and go to bed, I managed to track it down. If you have an anchor (<a>) tag that contains block-level elements (which is technically not allowed, but all other browsers seem to be ok with it), such as <p> or <div>, the anchor will close itself before the elements it is supposed to contain. E.g.: <a> <p>hello</p> </a> Will render as: <a> </a> <p>hello</p> If you give the anchor an href, though, even if it is just a blank string (<a href="">...), it renders fine. Here’s a page that demonstrates the bug: http://footle.org/misc/opera-bug.html. Anyway, I filed a bug with the Opera folks. Time for bed. [read]
Ok, thanks to Google for putting their dictionary out there, which I wrote about a couple of posts ago, but the quality of the results pales in comparison to Wordnik, which I’ve just discovered. It won’t do language translation for you, but as far as the English language is concerned, this is pure word porn. So here’s an updated bookmarklet for you: wordnik Just drag that to your bookmarks bar, select a word or phrase on the page, and click. Happy bonus: just clicking on the bookmarklet without highlighting a word first will give you a random word (thanks to Wordnik, not me). Cool! [read]
One of the security measures we have in place at Wesabe is to invalidate a session and log the user out if their User-Agent or IP address subnet changes between requests. However, we recently had a number of complaints that users were getting logged out prematurely. Looking at one user’s session in the logs, I noticed that their user agent string had “chromeframe/4.0” appended for some requests, but not others. It turns out that Google’s Chrome Frame only modifies the user agent for top-level requests, and not for subsequent requests sent as that page loads: It was a compromise to keep code complexity down. In order to tag every request from IE...we have to have hooks in place at many more places (not to mention supporting the different things IE6, 7 and 8 do). So, as a compromise we decided to keep it down to only the single hook that allows us to tag top level requests. This fix on our side was to just strip the chromeframe identifier from the user agent string, although it makes me wonder what other browser extensions cause similar issues and whether invalidating a session when the user agent changes is even a worthwhile security measure. [read]
Google recently launched a decent dictionary, so for you logophiles, here’s a bookmarklet that will bring up the definition of whatever text you have selected on the page: gDefine. Just drag it to your bookmarks toolbar, then double-click a word somewhere and click the button to get a definition. It is set to use the English dictionary, but if you want to use it in a different language (or even translate between languages), just change the langpair parameter from en|en to, say, fr|fr for French, de|de for German, etc. (try a search on the Google Dictionary site in the language you want and look to see what the langpair parameter is in the url). Also, the hl parameter controls the language of the results page, so if you want your results to be in that language as well (as opposed to English), change that accordingly. Here are a few to get you started: gDefine (French) gDefine (Spanish) gDefine (German) And translations: gTranslate (French>English) gTranslate (English>French) Enjoy! [read]
cyoa: Impressive analysis of Choose Your Own Adventure books, a fixture of my childhood. [read]
There's a weekly Tuesday evening Ultimate Frisbee game at a field near my house (sadly, I haven't gotten out to play in ages). Now that it's getting dark early, they need the lights on in order to play. The city is usually pretty stingy with the ... [read]
A literary appreciation of the Olson/Zoneinfo/tz database ë Jon Udell [read]
The Billion Dollar Gram | Information Is Beautiful: Billions spent on this. Billions spent on that. It's all relative, right? [read]
While poking around the database in my rails console, I often found myself jumping to the mysql console just so I could get an easier-to-digest view of the data. I finally had enough of this silliness and wrote up a quick function to dump of set of ActiveRecord objects in the mysql report format. [read]
A colleague was asking today how he could see if there’s data waiting on STDIN in ruby (on Linux). On OS X, this is pretty straightforward: if $stdin.stat.size > 0 puts "got something: #{STDIN.read}" else puts "nada" end That doesn’t work in Linux, though. After some digging, I found this post, which lead to: require 'fcntl' flags = STDIN.fcntl(Fcntl::F_GETFL, 0) flags |= Fcntl::O_NONBLOCK STDIN.fcntl(Fcntl::F_SETFL, flags) begin puts "got something: #{STDIN.read}" rescue Errno::EAGAIN puts "nada" end Which works on both platforms, but is (a) ugly (catching an exception), and (b) requires you to actually try to read from STDIN. In playing around with that, though, I noticed that STDIN.fcntl(Fcntl::F_GETFL, 0) returned 0 if there was something on STDIN, and something non-zero (2 in OSX and 32770 in Linux) if STDIN was empty. So now the code is simple again: require 'fcntl' if STDIN.fcntl(Fcntl::F_GETFL, 0) == 0 puts "got something: #{STDIN.read}" else puts "nada" end I’m not sure how reliable that is on other platforms (it won’t work on Windows—fcntl is a Unix-y thing). If I’m understanding fcntl.h correctly, a return value of 2/32770 (0x8002) seems to indicate that the io stream in question is open for writing. I’m not sure if that is intended or just a side-effect. Anyone know? [read]
One of the more annoying things about Leopard is that when you plug in an external monitor that is set up to be your primary, your open applications don’t move over to the monitor. I was having to manually drag everything over. Then I realized that if I hid all my apps before plugging in the monitor, when I brought them up again, they’d be in the right place. So now the question was: is there an easy way to hide all your apps without cycling through them all? It took a bit of digging, but it turns out there is. Cmd-option-click on an app in the Dock and it will hide all other apps. So I tried Cmd-option-click on the desktop itself and…voilà! Update: Technically, you’re Cmd-option-clicking on the Finder, so if you have any Finder windows open, they won’t hide. Bummer. [read]
I hadn’t posted this before because it seemed kind of trivial and silly, but I used it again the other day and decided maybe I should share the joy after all. Far too often I find myself jumping through hoops building ActiveRecord find conditions arrays dynamically depending on what arguments were passed in to the method. So I cooked up a simple little class to clean that up a bit: # class to construct conditions for AR queries # cc = ConditionsConstructor.new('foobar > ?', 7) # cc.add('blah is not null') # cc.add('baz in (?)', [1,2,3]) # cc.conditions # # => ["foobar > ? and blah is not null and baz in (?)", 7, [1, 2, 3]] class ConditionsConstructor def initialize(str = nil, *args) @conditions_strs = str ? [str] : [] @conditions_args = args end def add(str, *args) @conditions_strs << str @conditions_args += args end def conditions [@conditions_strs.join(' and ')] + @conditions_args end end [read]
I'm pretty late to the party, but I just ran across this on Chuck Hoffman's blog: "Grab the nearest book, open to page 123, go down to the 5th sentence, and type up the 3 following sentences." Before I do that, I thought it would be interesting to try to track this meme back as far as possible. For the sake of my sanity, I'm not following multiple paths from posts that link to more than one source--I'll just pick the link that seems to be the primary source. So here goes: nothing happens linked to Knowing and Doing linked to Exploration Through Example linked to a thaumaturgical compendium linked to michaelzimmer.org linked to Chronicles of Dissent linked to Fergie's Tech Blog linked to Threat Level linked to Danger Room linked to Abu Muqawama linked to Afganistanica: Mountain Tourism in Afghanistan linked to [My] State Failure Blog linked to Kings of War linked to MountainRunner linked to zenpundit.com linked to the glittering eye who gave a bad link that I had to track down to Independent Liberal linked to Donklephant linked to Stubborn Facts linked to another post on the same blog, linked to Sideways Mencken, who did not link, but referenced Internet Ronin linked to AmbivaBlog [and now I'm starting to tire of this game] linked to The Anchoress linked [not directly, had to dig it up *sigh*] to Some Have Hats linked to [again not directly...grr] Church of the Masses linked [incorrectly] to The Daily Grotto [warning! annoying chanting] linked to Aussie Coffee Shop linked to Where Angels Go which mentioned, but failed to link to the person who tagged them, so it's a dead end. Whew. This was getting a bit tedious. Not to mention that it seemed to have gotten firmly entrenched in religious/pro-life blogs, which were getting on my nerves. Just as well. Turns out Google returns almost half a million hits for "page 123" sentence, and there doesn't seem to be a way to sort results by date (the daterange operator failed me completely). The oldest post I could find on Google Blog search was this one, from November 24, 2004 (albeit for a slight variation on the meme--just the fifth sentence, not the three following it), but which clearly isn't the origin. Seems it was a meme floating around on LiveJournal around then. If anyone has better search-fu and can find an earlier post, let me know. At this point, I've made myself pretty sick of this meme. Often it appears to be a means by which people can brag about whatever erudite text they may or may not actually be reading. Even so, I can't have wasted this much time and not play along. I've got two books nearly equidistant from my keyboard now. One is, at least in my field, too mundane to bother with: the Pickaxe book. The other sports the following 123/5/3 sentences: The latissimus dorsi muscles have a major role in the deadlift: from the floor, the lat pulls back on the humerus to keep the arms from swinging forward away from the shins, and acts as an anchor on the upper part of the humerus to maintain the position of the bar directly under the shoulder blades until the bar crosses above the knees. The lats act in an essentially isometric way from the floor to the point where hip extension allows the arms to become vertical. At this point tension comes off the lats, and as the back becomes vertical, the arms drag the bar into the thighs as they assume an angle behind the vertical, opposite the starting position. That's from Starting Strength by Rippetoe and Kilgore, which is a great book for learning how to do basic barbell lifts correctly (something we do a fair bit of in Crossfit). Coincidentally, that's just a couple pages past where I was actually reading about the deadlift. (Most interesting tidbit I've learned so far from the book: the importance of a solid grip when doing the dealift. If your hands start slipping, proprioceptive (love that word) feedback tells the back that what you're trying to lift is too heavy, and it basically gives up.) I'm not going to inflict this on anyone else, but if you're reading this, feel free to prate on about whatever book you have near you. [read]
I recently checked out the Scramble application on Facebook, which is a knock-off of Boggle. It’s a fun diversion, even though I usually get my ass kicked. There’s an annoying chat window there, too, and invariably someone will accuse someone else of cheating, at which point a number of people will pipe up with “how would you cheat?” This got me thinking…how would you cheat? You’d just have to write a Scramble puzzle solver that spit out all the words in on the board. That sounded like a fun problem. I hadn’t done anything with recursion in a while, so I gave it a go. Below is what I came up with. You’ll need a word list to run it. I originally used /usr/share/dict/words, but it was missing a lot of valid words (I think it only has base words, no plurals or tenses). So I stole the scrabble word list from the Debian scribble package, which worked much better. Anyway, here you go: boggler.rb words.gz I did try this out a couple times on Scramble and won handily. There’s really not much fun in that, though. Note: This was really just an exercise in coming up with a fun algorithm, and is primarily intended for programmers. If you’re just looking to cheat at Boggle/Scramble, there are plenty of web sites that will help you with that, and with a lot less effort. [read]
"dolphins" by talkrhubarb I thought I knew a fair bit about MySQL before I started working on Wesabe, but I was wrong. Once your tables start to get really massive, all sorts of fun issues come out of the woodwork. There are a thousand articles out there on the basics of database optimization (use indices, denormalize when necessary, etc.), so I'm not going to cover those again. Here are some tips--most specific to MySQL--that I've learned on the job: MySQL only uses one index per table when doing a query, and the query optimizer doesn't always pick the best column, so indexing everything is not necessarily a great idea. If you're always selecting on a certain set of columns, create a single multi-column index related to the above, don't put indices on columns with a low cardinality. You can check the cardinality of a column by doing SHOW INDEX FROM table_name if any text or blob fields are part of your SELECT query, MySQL will write any temp tables to disk rather than memory. So where possible, use large VARCHAR fields (on 5.0.3 or higher) instead. See: http://railspikes.com/2007/9/14/one-reason-why-MySQL-sucks if you need to do case-sensitive matching, declare your column as BINARY; don't use LIKE BINARY in your queries to cast a non-binary column. If you do, MySQL won't use any indexes on that column. SELECT ... LIMIT offset, limit, where offset is a large number == bad, at least with InnoDB tables, as MySQL actually counts out every record until it gets to the offset. Ouch. Instead, assuming you have an auto-incremented id field, SELECT * FROM foo WHERE id >= offset AND id < offset+limit (props to Coda for that one). Update: If you have a query in which you're looking for the most recent n items of something—e.g. SELECT * FROM foo WHERE bar = ? ORDER BY created_at DESC LIMIT 10)—your query will be significantly faster if you create a multi-column index on whichever field(s) you're selecting on plus the timestamp field that you're ordering by (in this case, CREATE INDEX idx_foo_bar_created_at ON foo (bar,created_at)). This allows MySQL to look up the records directly from the index, instead of pulling all the records where bar = ?, sorting them (probably first writing them to a tempfile), and then taking the last 10. EXPLAIN is your friend. If you're seeing "Using temporary table, Using filesort" in the "extra" section of EXPLAIN, that's bad. I’ll add more as we come across them. If you have any other non-obvious tips, leave them in the comments or email me (brad@ this domain) and I’ll add them too. By the way, an invaluable book for learning how to squeeze the most out of MySQL is High Performance MySQL, by Jeremy Zawodny and Derek Balling. [read]
I decided to give tumblogging a try. I’m hoping that ease of use and lowered expectations will get me to post more often, and I’m liking it so far. It covers the large middle ground between blogging and twittering. I can put up something interesting without feeling like I need to construct a coherent post around it, and I don’t feel like I’m pestering people with twitters (not to mention that it allows for richer media types). I realize I could just do the same thing on this blog if I wanted to, but there’s something about the minimalist formatting and ease-of-use that just seems to work. So I’ll continue to post longer pieces here (I’ve got a number that I need to just sit down and write), and I’ll keep posting inane comments to Twitter, but I imagine that the majority of interesting things I find will end up on my tumblelog. [read]
Chris Wanstrath made a perfectly reasonable request earlier today, so I came up with the following Greasemonkey script: // ==UserScript== // @name Social Graphs Must Die // @namespace Footle // @description Redact all mentions of "social graphs"; // ==/UserScript== document.body.innerHTML = document.body.innerHTML.replace(/(social\s+graph(s?))/ig,"<span style='background:#000;color:#000'>$1</span>"); Actually closing a tab (that wasn’t opened by Javascript) in Firefox 2 isn’t possible AFAICT, but this is more fun anyway. [read]
After hearing a couple complaints from friends about the amount of spam they’re getting, I decided to take a quick look to see where I stood. At the risk of setting myself up, I should say that I get no spam (channeling John C. Dvorak). Or at least very little–maybe one every other day gets through to my inbox. I think the trick is just to have multiple levels of filtering. I use a Gmail address for most online transactions (for online transactions that I really don’t care about–if I’m doing a one-time registration just to get a trial key, for example–I use my Yahoo address, which is nothing but a spam bucket). My Gmail address gets forwarded to my personal email address (@footle.org), which has SpamAssassin running on the server. If it makes it past SA to Mail.app, it gets inspected by SpamSieve, an awesome Bayesian spam filter for OS X mail clients. Anyway, my stats for the last 24 hours: Server-level filters (spam caught): SpamAssassin: 281 Gmail: 32 Client-level filter (spam caught): SpamSieve: 17 Spam getting to my inbox: 0 Non-spam messages: 52 False positives: 1 (just Flavorpill, so not a big deal) (Wow…330 pieces of spam in a single day. That seems way worse than when I last checked.) [read]
Bonjour-enabled Git. (Although Coda tells me that Dave F. at Powerset already approached him with this idea. Neither of us have time to work on it.) I started using Git recently and although it’s a bit confounding at times, I love how cheap and easy branching is, and that you can commit incremental changes to your local repository, then later commit them in batch to subversion. Git allows for distributed development, but unless you’re on the same server as other developers, or you go through the trouble of mounting their repository over the network, you can’t check out their repository. Seems like it shouldn’t be too difficult to give Git Bonjour powers. Update: Chad Fowler has started such a project. [read]
We recently released a very slick Firefox extension at Wesabe. It was written by my colleague Tim Mason, but I helped figure out one small piece of it—namely, how to do binary multipart POSTs in Javascript—and since it involved many hours of hair-pulling for both of us, I thought I’d share the knowledge. (Tim says I should note that “this probably only works in Firefox version 2.0 and greater and that it uses Mozilla specific calls that are only allowed for privileged javascript—basically only for extensions.”) One of the cool features of the plugin is the ability to take a snapshot of a full browser page and either save the snapshot to disk or upload it to Wesabe (so you can, for example, save the receipt for a web purchase along with that transaction in your account). The snapshot is uploaded to Wesabe via a standard multipart POST, the same way that a file is uploaded via a web form. Tim was having trouble getting the POST to work with binary data at first, and he had other things to finish, so he wanted to just base-64-encode it and be done with it. I was reluctant to do that, as the size of the upload would be significantly larger (about 137% of the original). Also, Rails didn’t automatically decode base-64-encoded file attachments. But Tim had other bugs to fix, so I submitted a patch to Rails to do the base-64 decoding. I was pretty proud of this patch until it was pointed out to me that RFC 2616 specifically disallows the use of Content-Transfer-Encoding in HTTP. Doh. They also realized that it is a colossal waste of bandwidth. Since Tim was cramming to meet a hard(-ish) deadline set for the release of the plugin, I offered to lend my eyeballs to the binary post problem. This could be a very long story, but I’ll just get to the point: you can read binary data in to a Javascript string and dump it right out to a file just fine, but if you try to do any concatenation with that string, Javascript ends up munging it mercilessly. I’m not sure whether it is trying to interpret it as UTF8 or if it terminates it as soon as it hits a null byte (which is what seemed to be happening), but regardless, doing "some string" + binaryData + "another string", as is necessary when putting together a mutipart post, just does not work. The answer required employing Rube Goldbergian system of input and output streams. The seed of the solution was found on this post, although that didn’t explain how to mix in all of the strings needed for the post and MIME envelope. So here it is, in all it’s goriness: Update: Spaces removed from multipart boundary per Gijsbert’s suggestion in the comments (thanks!). [read]
I ran across a bug in Date::Format today, and after spending a few hours hacking away at a fix (the date/format.rb code is uuuuugly and sloooow…someone should really rewrite that. Better yet, rewrite it in C), I thought I’d submit a patch. So I grabbed the ruby_1_8 branch and lo and behold, my issue had already been fixed! So the question now was how to monkeypatch the entire Date::Format module? Simply require-ing it as a plugin doesn’t work, since Date::Format is already loaded at that point. The trick then is to use load instead of require. First, I tried this: load 'date/format.rb' (note: load needs the actual filename; it doesn’t have the magic that require does) but that gave me an error: in `load': wrong number of arguments (1 for 0) (ArgumentError) Turns out Rails::Plugin::Loader defines its own load, so: Kernel::load 'date/format.rb' et voila! [read]
It took me a bit of digging to find this, so I thought I may as well spread the word here. I’ve started using Adobe Lightroom for managing my photos, but every time I connect my camera or card reader to import photos, iPhoto would launch, and I couldn’t figure out where I could change that. I finally discovered that the setting is in the Image Capture application. Run that, then go to Preferences and you’ll find it there. Why they wouldn’t also put it in iPhoto, Lightroom, or even on a preferences panel is beyond me. [read]
Marc and I presented our Super Ninja Privacy Techniques for Web App Developers talk again yesterday at Web 2.0 (we also did it at ETech last month), and it seemed well received. I still need to work on my presentation skills; I’m not an old hand at this like Marc is. I’m working on a follow-up to my privacy wall post which will describe a much better way to go about keeping a user’s private data private, using an encrypted “Locker”. I’ll also go into detail about how we deal with password recovery. Hopefully by preannouncing this, I’ll force myself to get off my duff and finish the post. So stay tuned, and bug me if I don’t have it up soon. [read]
Google recently announced that it will soon start anonymizing search logs older than 18-24 months. Full details can be found in their Log Retention Policy FAQ (PDF). This is a heartening step back towards their “Don’t Be Evil” corporate philosophy, which some think has been largely abandoned. I’ve just recently started using Scroogle as a way of defeating their tracking of my every search (their site is awful; Wikipedia has more readable information about the project), although the motives of the man behind it, Daniel Brandt, who also runs the Google Watch site, may be questionable. Still, he doesn’t have much incentive for keeping a log of queries and IP addresses, and if he did, since he’s not giving me a cookie, he can’t tie all my searches together. [read]
My colleague Coda Hale just released a sweet bcrypt-ruby gem that does password hashing right. It also provides for future-proofing by enabling you to assign a computational cost for generating a password hash and letting you version your password hashes. You have no more excuses. [read]
Just Another Brick In The Wall?by Iain Cuthbertson We deal with a lot of very private data at Wesabe, so security and privacy are our top concerns. In this post I will describe one of our primary means for assuring privacy, a technique that is general enough that any site can use it. Our creative name for this technique is the privacy wall. Later, I'll go on to tell you ways to hack the wall, just so you don't get too comfortable. The Privacy Wall The idea is simple: don't have any direct links in your database between your users' "public" data and their private data. Instead of linking tables directly via a foreign key, use a cryptographic hash [1] that is based on at least one piece of data that only the user knows—such as their password. The user's private data can be looked up when the user logs in, but otherwise it is completely anonymous. Let's go through a simple example. Let's say we're designing an application that lets members keep a list of their deepest, darkest secrets. We need a database with at least two tables: 'users' and 'secrets'. The first pass database model looks like this: The problem with this schema is that anyone with access to the database can easily find out all the secrets of a given user. With one small change, however, we can make this extremely difficult, if not impossible: The special sauce is the 'secret_key', which is nothing more than a cryptographic hash of the user's username and their password [2]. When the user logs in, we can generate the hash and store it in the session [3]. Whenever we need to query the user's secrets, we use that key to look them up instead of the user id. Now, if some baddie gets ahold of the database, they will still be able to read everyone's secrets, but they won't know which secret belongs to which user, and there's no way to look up the secrets of a given user. Update: A commenter on the Wesabe blog brought up the important point of what you do if the user forgets their password. The recovery method we came up with was to store a copy of their secret key, encrypted with the answers to their security questions (which aren't stored anywhere in our database, of course). Assuming that the user hasn't forgotten those as well, you can easily find their account data and "move it over" when they reset their password (don't forget to update the encrypted secret key); if they do forget them, well, there's a problem. Attacking the Wall I mentioned earlier that you store the secret key in the user's session. If you're storing your session data in the database and your db is hacked, any users that are logged in (or whose sessions haven't yet be deleted) can be compromised. The same is true if sessions are stored on the filesystem. Keeping session data in memory is better, although it is still hackable (the swapfile is one obvious target). However you're storing your session data, keeping your sessions reasonably short and deleting them when they expire is wise. You could also store the secret key separately in a cookie on the user's computer, although then you'd better make damn sure you don't have any cross-site scripting (XSS) vulnerabilities that would allow a hacker to harvest your user's cookies. Other holes can be found if your system is sufficiently complex and an attacker can find a path from User to Secret through other tables in the database, so it's important to trace out those paths and make sure that the secret key is used somewhere in each chain. A harder problem to solve is when the secrets themselves may contain enough information to identify the user, and with the above scheme, if one secret is traced back to a user, all of that user's secrets are compromised. It might not be possible or practical to scrub or encrypt the data, but you can limit the damage of a secret being compromised. My colleague and security guru Sam Quiqley suggests the following as an extra layer of security: add a counter to the data being hashed to generate the secret key: secret key 1 = Hash(salt + password + '1') secret key 2 = Hash(salt + password + '2') ... secret key n = Hash(salt + password + '<n>') Getting a list of all the secrets for a given user when they log in is going to be a lot less efficient, of course; you have to keep generating hashes and doing queries until no secret with that hash is found, and deleting secrets may require special handling. But it may be a small price to pay for the extra privacy. Finally, log files can be a gold mine for attackers. There's a very good chance you're logging queries, debug statements, or exception reports that link users to their keys or directly to their secrets. You should scrub any identifying information before it gets written to the log file. So That's It, Right? The privacy wall is far from a silver bullet. Privacy and security are hard—really hard—particularly so if your app is taking private data and extracting information out of it for public consumption, like we are at Wesabe. The privacy wall is one of a number of methods we're using to insure that our users' private data stays that way. If you're lucky enough to be going to ETech next month, definitely check out Marc's session on Super Ninja Privacy Techniques for Web App Developers. I hope you found this helpful. Let me know what you think; I appreciate any and all feedback. And if you've got any cool privacy techniques up your sleeve, share the knowledge! <p>[1] A cryptographic hash is way of mapping any amount of plain text to a fixed-length “fingerprint” such that the same text always maps to the same hash, and given a hash, it is impossible to generate the text from which it was derived. Hashes are wonderful things with many uses. If you’re a developer, and you didn’t already know this, stop reading now and go here or here, and learn how to generate a SHA1/2 hash in your programming language of choice. Come back when you’re ready. I’ll wait.</p> <p>[2] You can throw in a salt too, to be safe; just make sure that you’re not using the same hash that you’re using for checking the user’s password. You are smart enough not to store passwords in plaintext in the database, aren’t you?</p> <p>[3] Danger, Will Robinson! Keep reading.</p> [read]
We live in Campfire at Wesabe, so naturally we've developed a number of tools and hacks around it. One of these days I'll get around to writing about the Campfire bot framework we've developed, but right now I'll start with something simpler: a way to upload screenshots to Campfire with only a few keystrokes (riffing of my colleague Coda Hale's desire to eliminate the need for a mouse). Note that this is for Mac users only, so if you're one of the poor souls who hasn't seen the light yet, you can stop reading now. Two prerequisites (other than a Mac): Quicksilver and Pyro. Quicksilver should be familiar to most Mac users already. It is an incredibly powerful application that can be quite obtuse but can save you gobs of time if you learn how to use it. Pyro is a client for Campfire. Why do you need a client for a browser-based chat app? Getting message notifications in your dock is one reason, but the biggest reason is that there seems to be a memory leak in Firefox that causes it to grind to a halt if you've had Campfire up for too long. But I digress. You first need to enable the Screen Capture Actions plugin in Quicksilver (go to Plugins -> Recommended, and check Screen Capture Actions). Then to go Catalog -> Quicksilver and make sure that Internal Commands is checked). This should give you Capture Region, Window, and Screen commands. Next, the secret sauce: a bit of AppleScript that uploads a file to Campfire via Pyro: on open theFile tell application "Pyro" upload theFile to room "[your campfire room name]" in campfire "[your campfire].campfirenow.com" end tell end open Paste that into Script Editor and save it as PyroUpload in ~/Library/Application Support/Quicksilver/Actions (if this folder doesn't exist, create it), and restart Quicksilver.Now you can upload a screenshot with just Cmd - Space - [Capture Window|Region|Screen] - Return - [take your screenshot] - PY - Return. Have fun! [read]
I’ve been too tired today to do any actual work, so I’ve spent much of the day camping out on the Wesabe site stats. It’s terribly exciting having so many people hitting your site and so many signing up (almost 1/3 of our unique visitors have created accounts). The site has been humming along beautifully, too–major props goes out my colleague Coda Hale for his Apache/Mongrel/Pen prowess. [read]
The reason I’ve had my head in the sand for the last six months is that I’ve been working on a new startup, and we’ve finally launched publicly. Check us out. Here’s a screencast demo of the service: </param> </embed> Let me know what you think. (Or better yet, send your thoughts to support@wesabe.com.) [read]
<p>No, I haven’t abandoned this site. I’ve just been working my butt off at a new startup. I’ll try to get some content up soon, but in the meantime, enjoy some pictures of the newest member of our family. </p> [read]
I am far more productive when writing Rails code than when writing Java. I just realized that one of the reasons for my lower productivity in Java is the need to recompile every time a make a change to a page on the site. In the 15 seconds or so it takes to recompile and redeploy to Tomcat, I get bored and am apt to go check my new favorite news site, popurls, or my RSS feeds, or (less likely) post to my blog. Suddenly those 15 seconds have become 5 minutes. And this happens many times throughout the day.With Rails, I make a change, refresh my browser, and there it is. On to the next step. [read]
It took me forever to figure this one out, but if you want HTTP Authentication to work with Apache 2 and mod_fastcgi, you need this in your apache conf file: [read]
While researching the best way to handle calling an external program from Ruby (and capturing stdout & stderr), I came across this post, which is a good review of how processes and threads work: http://www.ruby-forum.com/topic/65155#75363 I still haven’t figured out exactly how I’m going to do this, but I’ll post it here when I figure it out. Ruby has a few different ways of opening and communicating with processes, but all seem to be lacking in some way or another. IO.popen lets you write to the process’ stdin, and read from its stdout, but you can’t get stderr without jumping through serious hoops (like redirecting stderr to a file and then reading the file…ugh). Open3.popen3 (brilliant naming) gives you stdin, stdout, and stderr, but the subprocess runs as a grandchild, so there seems to be no way to wait for it to finish. [read]
There’s an interesting series of posts over at O’Reilly Radar. Tim O’Reilly asked people how they were using databases in their “Web 2.0” applications (although I think the Web 2.0 part of it is for the most part irrelevant). The responses so far have made for interesting reading. So far we’ve heard from Second Life, Bloglines and Memeorandum, Flickr, NASA World Wind, and craigslist. One of the lessons learned is that with a high-traffic site, at some point you have to break your database up so that the “hot” data is spread across a number of boxes. This got me thinking. It should be possible to build a tool that analyzes your database usage and, given a number of slave boxes to configure as it sees fit, automatically configures masters and slaves and distributes your data across those boxes as necessary. This would not be a one-time only process either; it would continue to monitor usage and performance and adjust accordingly. Certainly not an easy task, but should be doable. [read]
I use iTunes both on a Mac and Windows. In OS X, Command-H hides an app, something I do so often I just do it without thinking. However, in iTunes on Windows, Control-H (aka backspace) deletes whatever object you’re on, with no undo. I just deleted a playlist I had spent a long time working on and I see no way to get it back. Why don’t I at least have the option to undo? Terrible UI. Surprising that Apple of all companies made such a basic mistake. [read]
Amazon just launched a new service, S3 - Simple Storage Service. It is a web service that allows you to store as much data as you like, with file sizes up to 5GB, and you just pay for the storage you use and the data transferred. Rates are very reasonable, too – $0.15/GB/month of storage, and $0.20/GB in data transferred.This is pretty interesting. It gives developers the ability to create applications requiring significant storage space without having to make a huge upfront investment in equipment and expertise. Want to write your own Flickr? Go for it. Granted, it’s risky relying on a third party for a core part of your business, but you only need them until you get your million users and can get enough funding to build your own storage backend.Google is apparently working on their own storage backend, Google Drive. It will be interesting to see how this plays out. Nothing but good news for aspiring entrepreneurs, though.via TechCrunch [read]
One of the most exciting new technologies I’ve seen in a long time: a multi-touch touch screen. The video says it all. </embed></param> [read]
John C. Dvorak has a good piece at pcmag.com about speculations that Google is going to be creating their own network (see the “Google is the Internet” scenario from the article linked in my previous post). I certainly hope they do, as telcos have been dragging their feet on broadband for a long time, and acting like the Mafia whenever someone encroaches on “their” territory. More articles on the subject: Rumours mount over Google's internet plan Google to Telcos: Who Needs You? [read]
Great article from Business 2.0 describing four future scenarios for Google:Imagining the Google Future [read]
My imsmarter proxy stopped working last week, and I just got around to going to their site to see what was up. Looks like they’ve shut down. A bit of a pity; I thought it was a useful service. I use IM on four different machines and it made finding something from a past conversation a lot easier. Actually, though, what I used most often was its reminder feature. I could send it an IM saying “Remind me in 2 hours to check the car” and it would do just that, saving me many parking tickets. I imagine there are other services like that out there; I should check around. Actually, that would be a pretty trival thing to implement myself. Hmmm. [read]
Your phone records are for sale“The Chicago Police Department is warning officers their cell phone records are available to anyone – for a price. Dozens of online services are selling lists of cell phone calls, raising security concerns among law enforcement and privacy experts.” [read]
Chris Anderson of Wired / The Long Tail has started a wiki page to track public blogs by Fortune 500 companies. The list isn’t terribly long yet, but I’m sure it will be growing, both as more people discover existing company blogs and as more companies jump on the bandwagon. Speaking of which, we’ve jumped on the bandwagon ourselves at Triporama. The Triporama Blog isn’t yet linked in from the main site (it will be soon), but Wendell has already posted a great piece about the origins of Triporama. Happy New Year! [read]
I’ve been pretty busy lately, if the infrequency of my posts is any indicator, but it’s paid off: Triporama officially launched yesterday. We sent out some 300 emails and then immediately left to go to the bar. Fortunately, the site held up, with only one serious bug so far, which I fixed last night. Not that I can slack off now…we’ve got a mile-long list of features we’d like to implement. It’s great to finally get it out there, though. [read]
This: Top 10 Innovative Web 2.0 Applications of 2005 led me to this: TagCloud which let me to this: Term Extraction Documentation for Yahoo! Search Web Services Now to just figure out some cool things to do with it. [read]
Clay Shirky has a great piece about ontologies, categorization, and how they don’t really make sense for most of the Web today: Ontology is Overrated Paul Graham’s new essay tries to make some sense of everyone’s favorite buzzword: Web 2.0 Happy Thanksgiving! [read]
Video of Dan Osman speed-soloing up Bear’s Reach at Lover’s Leap (near South Lake Tahoe, CA). Need I say don’t try this at home? http://www.ebaumsworld.com/video/watch/1220/ [read]
I'm posting this entry from within Flock, a new browser based on Mozilla, the codebase which underlies Firefox. You might think the world needs a new browser like it needs a hole in the head, and I more or less agree with you, but Flock has some really cool features: You can link your bookmarks to your del.icio.us account so that they're always available when you move between computers. It integrates with a number of blogging services and software to let you blog directly from the browser. You can even highlight a chunk of text on the page, right-click on it and select "Blog This", and it opens up a blog post window with the text inserted and quoted, and the site name linked. (I'm finding that its blogging editor is incredibly annoying when you're trying to do bulleted lists) flickr integration, which seems to me to be a bit gratuitous, but it is kind of cool to get access to your photos in a toolbar (sorry, "topbar"). Integrated RSS feed reader, which right now has a really clunky interface. But this is just a developer preview release, so I'm sure that will improve. "The Shelf"...kind of a clipboard for web content. You can drag-and-drop links, chunks of text, or pictures on to it and later drag-and-drop them into your blog posts (it seems to be made pretty specifically for blogging; you can't double-click on items in your Shelf to open them up, and although you can drop them onto any application that accepts clipboard data, what you get is an HTML-wrapped version of your content). All in all, very impressive. I hope that some of these features (especially the social bookmarking integration) make their way over to Firefox, but I'll be keeping an eye on Flock's progress. [read]
Apple just released a new video-capable iPod, as was expected. You can also download music videos from the iTunes Music Store, at $1.99 a shot. That's cool. Yawn. But wait...what's really interesting is that you can also download episodes from some TV shows, like Lost or Desperate Housewives--also for $1.99. This is the start of a major paradigm shift. I'm not a huge TV watcher, but there are a few good shows out there that we've gotten hooked on (Lost, Smallville, Arrested Development, Battlestar Galactica). Not having cable and Tivo, though, I can't always catch my shows. I've always said I'd be happy to pay a small fee ($1.99 is about right) to be able to download an episode legally (it's not hard to find and download them "illegally" online). Now I can do that. Go Apple. The networks certainly weren't going to get their shit together to do it themselves. It's still got a long way to go. Only a few shows are apparently available. Actually, I don't know exactly what's available because the only links to videos on their "Music" store home page are for "Music Videos" and a big "Lost" graphic. Is it so hard to just have a separate video section in your store? I'm guessing that Apple tried to push this out the door quickly, before properly redesigning their store interface to support the videos. Or maybe not. It's not that hard to do. Regardless, I'm sure they'll sort it out soon enough. They're going to have to rename the Music Store, though. Media Store? Actually, my guess is that once they pad out their offerings a bit, they'll have a separate Video Store. An Apple Video Store will be huge for everyone involved. The networks will be able to squeeze new money out of shows that have long since been written off. You'll be able to relive your youth by downloading some Miami Vice episodes or "owning" your own copy (as much as Apple's DRM lets you own) of the classic "Master of my Domain" Seinfeld episode, not to mention getting access to all kinds of old shows that will never make it to DVD. And of course Apple get its cut. Go Apple. [read]
In lieu of anything else interesting to post, I'll clear out some of the links I've been saving: Writeboard I haven't had the opportunity to really use this yet, but it looks like a pretty cool collaboration tool. No more sending Word docs back and forth. It's also packaged with their equally cool Backpack app. Aurgasm I'm a bit late in the game here, but I've just started discovering the wonderful world of MP3 blogs. All sorts of great, eclectic music, for free. There are more MP3 blogs out there than you can shake your booty at, but this is one of my favorites. Improving Link Display for Print Absolutely ingenious. Using CSS & Javascript, you can automatically list all links on the print version of your web page as footnotes. I'm going to have to give it a shot on this site. Fog Creek Copilot For anyone who is, like I am, the person family and friends turn to when they have computer problems, this looks like a godsend for those trickier issues. Yes, computer remote control software is nothing new, but the really ingenious thing here is how easy it is to get someone set up. You just sign up with the site and give the person you're helping a code. They go to the site, enter the code, download the necessary software, and you're connected. [read]
We're in the middle of a fairly major site redesign at Triporama, and I was doing my best to have the layout be as purely CSS as possible, replacing the previous table-heavy layout. It's turned out to be a major headache, and I'm taking a step back now and reimplementing parts of the layout with tables. [read]
Spotlight was a great addition to OS X, and it works great for finding things buried in your filesystem. But for anyone used to Quicksilver or LaunchBar, it is painfully slow as an application launcher, which is really all I need most of the time. I recently reinstalled Quicksilver and I don't think I've touched Spotlight since then. [read]
Some random interesting stories/links I’ve run across in the last week or so: Photos from this year's Burning Man in the form of a graphic novel. Very nicely done. With an audio recording of you typing away at your computer for 10 minutes, it's possible to figure out everything you typed. (via Crypto-Gram) Tongue-eating bug found in fish. So gross, but so cool. (via Boing-Boing) Word Pads, a very addictive word game. My high score so far is only around 5000 18575. I didn't really discover this in the last week, but it's too cool not to get a mention: the Gmaps Pedometer (the link centers on a more interesting location than the default of Hoboken, NJ). It's a testament to the brilliance of the Google folks that people have been able to come up with great hacks like this. Finally, I'm going to try to play in the upcoming Ultimate Frisbee regionals, despite not having played since last winter, mostly due to my crappy back. I think I can hold it together for a weekend. Or at least a day. But this has inspired me: a 67-year old man comes out of retirement to play for his soccer team in their time of need. (via Fark) [read]
I'm not a C#/.Net programmer, but LINQ is too cool. You can basically use SQL-like querying on just about any collection, whether it be an array in memory, a database, and XML structure. This is one of those "why didn't anyone think of this before?" things. Yes, there have been similar tools for querying objects for a long time, but nothing this elegant. Watch the video for a quick (well, 30 minute) intro, or, if you prefer text, read the overview document. [read]
I share a hosted server with a few friends. We host a number of sites (including this one) and our own email. We have SpamAssassin set up and tuned pretty nicely (thanks, Jelo). Unfortunately, it's killing our server. SpamAssassin can suck up a lot of resources. Ideally we'd have a server dedicated to processing mail, but we can't afford that. So here's my idea: a spam filtering cluster. Have a bunch of [trusted] home-based servers set up with SpamAssassin and dynamic DNS. A mail comes into the main mailserver and if it isn't on a whitelist, it gets shipped out to one of the filter machines. The filter machine runs SA on it and lets the main server know the result (it doesn't need to sent the mail back, so no worries about low upload speeds). Emails over a certain size would probably just be processed on the main server to avoid the bandwidth and time required to send it out (large emails are rarely spam anyway). Finally, in order for the Bayesian filtering to function correctly, you'd have to periodically sync the data from the Bayesian learner out to the filter machines. I've looked at SpamAssassin a bit and I think it wouldn't be too hard. SA comes with a client and a server, spamc and spamd. Incoming mail gets piped through spamc, which takes care of shipping it off to spamd for processing. Spamd can live on any other host. It can also just report back whether the email was spam or not, rather than returning the entire email. So the only two things left to do are: Write a small client (spamb?) that maintains a list of hosts running spamd. When a message comes in, pick the next hostname in the list, and pass it on to spamc. Spamc sends it to the appropriate spamd host, and receives the response--either yes/no or in our case, the full SA report, which gets attached to the message headers. If we get the full message back, that means that spamc timed out trying to contact that spamd host. In this case, mark that host as being down, along with a timestamp, and take it out of the rotation for a certain period of time. A way to distribute the users' bayesian data files and prefs to the remote systems. Apparently spamd can read user from a SQL database, although I haven't looked into it to see if the bayesian learner data can be stored in a database. If so, that's an easy solution to the problem. Otherwise, you could just write a script that checks for changes in any of the user files and if it sees them, rsyncs them to each of the hosts. I'll let you know what happens if I get around to trying this out. [read]
I've been playing around with Ruby and Ruby on Rails lately. I'm not that far into it yet, but they both look interesting. I'll write more about my experiences later. In the meantime, if you're setting this up on a Mac (running Tiger) like I am, do yourself a favor and before you start getting into any of the Rails tutorials (like Rolling With Ruby on Rails), go here first: Ruby On Rails, Mysql, and OSX Tiger Woes Do what it says (including the fix it mentions), and you'll save yourself a lot of grief. Note that as one of the commentors mentions, it works just fine with mysql-ruby-2.7 also. Back to my tutorial... [read]
Just got an appointment reminder from Kaiser. Must remember not to put on my Parfum de la Fourmi. [read]
Some insight to be gleaned here: 10 Steps to a Hugely Successful Web 2.0 Company I don't completely agree with all of his "steps", particularly the examples he provides. Take #7--Get people hooked on free--for example. Yes, free will always win you fans, and a large user base has intrinsic value, but if you're providing a service that people are willing to pay for, by all means let them pay. (Regarding his Thefacebook vs. Match.com example--which one is actually making money?) Still, there are some good ones. My favorite: 6. Be mindnumbingly simple. Then again, this is a pretty age-old design mantra. [read]
I've had my web designer hat on lately. I've always been a much better judge of good design than a creator of good design, so I need all the help I can get. One thing that can make a huge difference in the quality of your site is color selection. Here are some sites I've been going to lately for chromatic inspiration. The Return of Design - Web Color Schemes A slew of pre-packaged palettes, all using web-safe colors (i.e. 256-color palette), each with a number of variations. ColorBlender Pick your starting color and ColorBlender will show you a 6-color palette for that color, and allow you to easily play with variations or customize them. You can also save them [to a cookie] so you have your personal blends available to you whenever you visit the site. You can also email direct links to your blend. Very cool. Color Scheme Generator 2 The ultimate color-generating tool. If you can't come up with a color scheme you like with this tool, you really are a lost cause. You can even see how your scheme looks to people with any one of eight different types of colorblindness. [read]
I just spend *hours* trying to get some CSS working only to discover that the reason it wasn't working was because I didn't have the DOCTYPE declared at the top of my html: <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "DTD/xhtml1-strict.dtd"> [read]
Today's diversion: Whose Fish? Took me about 45-50 minutes, the first 30 minutes of which were spent on a false start. After starting over with a better system for keeping track of everything, it wasn't that bad. I really doubt that this was Einstein's puzzle, and certainly the 2% figure is pulled out of someone's ass, but it's still good for some brain calisthenics. [read]
Last night Jacqueline was looking around online for flights to Seattle. She ended up falling asleep before she could book anything, but when she went to check again in the morning, she found that the same flights were about $30 cheaper. Perhaps it was just a coincidence, but my theory is that people/agents put holds on flights throughout the day, and those holds are released at the end of the day (assumedly midnight, although I don't know which timezone), freeing up seats and thus reducing prices. Has anyone else seen this? Or is this common knowledge and I've just been clueless? [read]
I finally caved in and signed up for Netflix, as you'll see if you look on the bottom of the sidebar. I've been a bit of Luddite, preferring to get my movies from one of the great local video stores. But we recently moved and now our closest video store is about a mile away--as opposed to about 200 feet away--so our movie-watching has dropped dramatically (while our watching of mind-numbing reality shows has increased accordingly). It seems to me that the queue is the key feature of Netflix. I can't count the number of times we've seen a preview for a movie and thought "We should rent that." Then you get to the video store and you spend half an hour walking up and down the Recent Releases trying to figure out what to get. Then again, sometimes you're in the mood for a particular kind of movie, and it might not be what's next in your queue. Hmmm...here's a business idea to help out all those struggling video stores in the new Netflix world: make an online service (let's call it FooFlix) where people can browse movies, get recommendations, and add movies to a queue, much like Netflix. But then install terminals in video stores and tie member's FooFlix accounts to their video store account. Then they scan/swipe/enter their store card/code in the terminal and it brings up the movies in their queue and highlights the ones that the video store has available. The stores would also be able to see what their customers (or an aggregate of all FooFlix customers) have in their queues, so they know what they should be ordering. As for a revenue model, you probably couldn't get away with charging customers for the service, but you could charge video stores to rent the terminal, and perhaps for customer queue data. You could probably do some pretty targeted advertising on the site, too. Probably not going to make me a billionare. If anyone steals the idea, at least send me a postcard or something. [read]
Joel Spolsky's latest article is largely a rehash of one of his earliest articles, whereby he expounds on his belief (which I share) that there's no substitute for top-notch programmers. The thing that got me thinking was his "shaky claim that most people are wrong in thinking you need an idea to make a successful software company," and this chart: Best Working Conditions => Best Programmers => Best Software => Profit! The problem I have with this is that you still need a pretty good idea to start with, or you're never going to get the best programmers in the first place. Great programmers aren't in it for the money; they want to work on stuff that interests them. Cool stuff. Sexy stuff. Having a great working environment with lots of other great programmers around is a big draw too, but how are you going to achieve that without the great idea in the first place? This hits a chord with me because I know a lot of great programmers with whom I would love to start a company, but I've been waiting for the perfect idea to come along and it hasn't. But perhaps the trick is to just come up with a good-enough idea -- good enough to get one or two other great programmers to dive into it with you -- and just take it from there. So...who's got a good-enough idea? [read]
Another great column by Mark Morford. Complaining about a bit of hidden sex in a game that glorifies sex, drugs, and violence does seem a bit misguided. Yeah, Puritan roots! [read]
Cool and a bit disturbing, and strangely addictive. Use your mouse to move her around. Pretty nice bit of programming. [read]
Somebody has deemed today Embrace Your Geekness Day. So what better way to celebrate than post to my blog. I searched around for information about the origins of this holiday, but found nothing other than lists of other obscure days (July 15th is Respect Canada Day!). How does one get a day proclaimed anyway? There are some openings left; one calendar I looked at had the 29th and 31st of this month still open. How about a Wear Flip-Flops to Work Day? (Googling for that came up with a single hit.) [read]
From a cookbook by the Ladies Auxillary of some hospital in Rhode Island: Scrumptions Muffins 1 cup softened vanilla ice cream 1 cup sifted self-rising flour Mix well. Pour into a muffin tin lined with paper cups, about 3/4 full. Bake at 350 F for 20 min. You know you want to make them. I know I will. Update: Made them. I had to improvise a bit, as I didn’t have self-rising flour. I added about 1/2 tsp. of baking powder instead. They were pretty bad. Very dense. Must have been the flour. [read]
Brad’s #1 rule of debugging web apps: always check your log files. All of them. [read]