Earlier this week Matt Buchanan ranted in Gizmodo about Consumer Reports’ review of smartphones. He didn’t read the actual reviews, of course. He just saw the top five list on someone else’s blog: the Samsung Blackjack II, the T-Mobile Wing, the Motorola Q9C, the T-Mobile Shadow, and the Blackberry Pearl Flip. Then he went to town.

Ignoring for the moment that four out of the five are Windows Mobile phones, they didn’t even pick new, actually good hardware. Not one of the phones, except for the Pearl Flip—which is actually the least capable phone in RIM’s new batch of devices—is even from this year. Its top phone, the BlackJack II, doesn’t even have Wi-Fi or a touchscreen, and is loaded with Samsung’s BS proprietary ports, rather than industry standard ones. Beyond that, where the hell are the other good smartphones? If they wanna be dated, where’s the BlackBerry Curve? They could shoehorn it in by reviewing one of the newer iterations with Wi-Fi. No Symbian?

In short, based on a top five list presented without context, Matt got angry that Consumer Reports didn’t pick the most recent smartphones and don’t get that the only thing that really matters about smartphones is the software.

Unlike Matt, I’m actually interested in context, so I dug up the actual article and ratings.
They judged the phones based on things like voice quality (both talking and listening), talk time, and ease of use. The article gives further sub-divided groupings of smartphones, like their recommended ones for multimedia (the iPhone 3G and the T-Mobile G1) or office-like tasks (the Samsung Blackjack II and the Motorola MOTO Q 9c). So it’s not just a top five list; it’s a constellation of recommendations based on usage.

Also unlike Matt, I understand the difference between generalist and specialist reviewers, the difference between tech-obsessed early adopters (hi, all of Gizmodo) and more average users, and the pitfalls of trusting numbers that are generated by qualitative rather than quantitative tests (such as “ease of use”).

Look, it’s simple: a generalist review site like Consumer Reports works best when you’re using it for items you’re not geekily obsessed about. Car enthusiasts bitch about their car reviews; audiophiles bash their stereo reviews. I’m sure if I were the kind of person to subscribe to Coffee Maker Magazine I’d be annoyed about their coffee maker reviews. And it’s okay to point out flaws in their methodology and places where you disagree with their results. But whinging about a top five list when you didn’t even do your homework and read the primary source? That’s as out-of-touch as Matt accuses Consumer Reports of being.