Does Search Ranking Matter?

Expedia A/B tested their hotel ranking algorithm against random ordering across 400,000 search sessions. The headline booking rate looked like a 620% lift, but it turned out to be a data construction artifact. The one clean metric, clicks per session, shows a small but real 2% reduction in the algorithm group. Whether that means better relevance or disengagement is something the data alone can't answer.