Milkshakes may not be as thrilling as that other "trial and error" product we discussed last week, but it’s another great example. Steve Levitt on the Freakonomics blog tells a story about he and his wife at a restaurant, where their menus were identical EXCEPT for the description of a milkshake. Sounds to Steve, and to me, like a good way to see which description sells more shakes. (Although if two people at the same table have different menus, how does the researcher know which customer saw which description? Makes much more sense to vary the menus from one restaurant to another, or from one day of the week to another.)
Internet marketers have learned this lesson pretty well. A google ad doesn’t have to land the clicker at the advertiser’s home page; more often, it lands the clicker at a "landing page" that sells a particular good or service. Well, the landing pages can be changed. One can simultaneously run two ad campaigns which send clickers on different landing pages, then test the success rate of the different pages. By this trial and error method, perfection is approached.
Or, you can assume that your high-paid advertising consultant knows it all, and do no experimentation at all.