This is the fifth in a series of posts about Paul Graham’s book Hackers & Painters.
I am a bit perplexed at “A Plan for Spam” (Chapter 8): In it, Graham describes Bayesian filtering, and his surprise at how effective it is compared to the more time-consuming alternative of building a complicated set of blacklists and whitelists. Because this method of spam protection requires a large corpus (ideally an individualized one for each user) of spam and non-spam messages, he writes, “each user should have two delete buttons, ordinary delete and delete-as-spam.” This is, of course, a feature familiar to anyone who’s used Gmail, Apple Mail or virtually any other recent e-mail client. I find it hard to imagine a world in which Bayesian spam filtering isn’t used, and I now receive very little spam in part because of it. So, this essay is completely right, but it feels like a relic from a bygone era when e-mail spam was a serious concern rather than a joke. Today, phishing is the much greater concern, and Bayesian filters are necessarily limited in their ability to stop e-mail designed to emulate missives from your bank or ISP. While “A Plan for Spam” may have been influential, it feels out of place in a book aimed at less transient concerns.
Yet taken as a complement to the following chapter, “Taste for Makers,” the spam essay conveys a subtle and important point: People tend to neglect a simple, elegant solution that requires a lot of thought up front (in this case, an understanding of Bayesian probability updating) when a more complicated, tedious, and less effective approach is available that requires little intellectual exertion (the blacklist-whitelist approach). The properties of the former solution are part of what Graham calls good design: “Good design is simple… Good design solves the right problem… Good design is hard… Good design looks easy… Good design resembles nature… Good design is often strange.” This last is interesting: “I’m not sure why,” Graham admits. “It may just be my own stupidity. A can opener must seem miraculous to a dog. Maybe if I were smart enough, it would seem the most natural thing in the world that eiπ = -1. It is after all necessarily true.”
The best advice is at the end of the chapter. “Intolerance for ugliness is not in itself enough,” Graham warns, “You have to understand a field well before you develop a good nose for what needs fixing. You have to do your homework. But as you become expert in a field, you’ll start to hear little voices saying, What a hack! There must be a better way. Don’t ignore those voices. Cultivate them.”
[Update: After writing this, I came across “Filters that Fight Back,” an essay Graham wrote in 2003 yet chose not to include in H&P. In it, he claims that spam could be fought more effectively if our e-mail clients automatically crawled suspect links, thereby imposing harsh bandwidth costs on spammers. It is a scheme that seems just crazy enough to work, and I’m a bit surprised that I haven’t heard of it being used in practice.]
0 responses so far ↓
Comments are closed.