AOL Does Not Understand the Trial and Error Economy

Fast Company magazine has a good article about AOL’s troubles.  There are plenty of troubles to chronicle, but here’s a particularly informative one:

In November [2006], search had posted 38.9
million visitors; by May, the figure was down to 35.2 million. [COO Ron] Grant
zeroed in on the multimedia character of AOL query results: If you
searched for "Radiohead," for instance, you would get not just text
articles on the band but also images, video, and links to their songs.
Grant saw this differentiation as a weakness — it slowed load time,
and the rich results meant that users were less likely to refine their
searches, thus delivering below-average page views. He ordered a change
to the page, making it look and operate exactly like Google’s. Yet it
turned out there were ways in which some users actually preferred the
old format. It was certainly different, offering people a reason to go
to AOL rather than Google. And the below-average page views could be
seen as a sign that users were finding what they wanted the first time
through. Also, according to former executives, the old search page
actually produced more revenue per search than Google’s.

In any case, changing the page backfired, badly. Search revenue fell
to $156 million, from $232 million the previous quarter. As a result,
AOL missed its revenue targets. "Management was blindsided by how
disruptive the change to search was," says Pali Research analyst
Richard Greenfield. "It’s troubling that they didn’t know what the
impact of the search change would be.

What do you do if you think that your search results page–a big key to revenues–needs to be revamped?  Those who understand the Trial and Error economy test the idea.  One way is to roll out the new version for a small subset of the users, say one percent.  With 35 million visits, one percent is a pretty big sample.  Give the guinea pigs three little buttons on the right hand side of the screen:

  1. I like it (just a vote, stay on the same page)
  2. I hate it
  3. I’d like to write a note about it.

That will get you valuable user feedback without jeopardizing your entire customer base.  I can imagine some folks liking the old screen, others liking the new screen, and some wanting to go back and forth.  That’s pretty danged easy; let people change their default search results type.  Let them also specify for a particular search that they’d like the alternative to their default.

The big dogs should understand this, but just in case they don’t, here’s what the new puppies should say:  "Great idea, boss.  Would you like me to work out a little test-of-concept experiment?"  That’s how to succeed in the Trial and Error economy.