My New Empirical Study on Defining and Measuring Search Bias

Josh Wright —  3 November 2011

Tomorrow is the deadline for Eric Schmidt to send his replies to the Senate Judiciary Committee’s follow up questions from his appearance at a hearing on Google antitrust issues last month.  At the hearing, not surprisingly, search neutrality was a hot topic, with representatives from the likes of Yelp and Nextag, as well as Expedia’s lawyer, Tom Barnett (that’s Tom Barnett (2011), not Tom Barnett (2006-08)), weighing in on Google’s purported bias.  One serious problem with the search neutrality/search bias discussions to date has been the dearth of empirical evidence concerning so-called search bias and its likely impact upon consumers.  Hoping to remedy this, I posted a study this morning at the ICLE website both critiquing one of the few, existing pieces of empirical work on the topic (by Ben Edelman, Harvard economist) as well as offering up my own, more expansive empirical analysis.  Chris Sherman at Search Engine Land has a great post covering the study.  The title of his article pretty much says it all:  “Bing More Biased Than Google; Google Not Behaving Anti-competitively.”

One clarification is in order.  The  Search Engine Land piece quotes Geoff responding to the author indicating that the research was undertaken independently.  It’s not clear from the way Sherman presents it in the article, but Geoff did acknowledge (as does the disclosure in the paper itself) that Google supports ICLE.  However, as Sherman notes, although the work was indirectly supported by Google, Google had no hand in the conception or execution of the project.  For that I’m the only one to blame, unfortunately.

Following is a summary of the study from the ICLE website.  I plan to blog about the results and their implications in the coming days in a series of posts.

Google has been the subject of persistent claims that its organic search results are improperly “biased” toward its own content.  Among the most influential is an empirical study released earlier this year by Benjamin Edelman and Benjamin Lockwood, claiming that Google favors its own content “significantly more than others.”  The authors conclude in their study that Google’s search results are problematic and deserving of antitrust scrutiny because of competitive harm.

new report released by the International Center for Law & Economics and authored by Joshua Wright, Professor of Law and Economics at George Mason University, critiques, replicates, and extends the study, finding Edelman & Lockwood’s claim of Google’s unique bias inaccurate and misleading. Although frequently cited for it, the Edelman & Lockwod study fails to support any claim of consumer harm — or call for antitrust action — arising from Google’s practices.

Prof. Wright’s analysis finds own-content bias is actually an infrequent phenomenon, and Google references its own content more favorably than other search engines far less frequently than does Bing:

  • In the replication of Edelman & Lockwood, Google refers to its own content in its first page of results when its rivals do not for only 7.9% of the queries, whereas Bing does so nearly twice as often (13.2%).
  • Again using Edelman & Lockwood’s own data, neither Bing nor Google demonstrates much bias when considering Microsoft or Google content, respectively, referred to on the first page of search results.
  • In our more robust analysis of a large, random sample of search queries we find that Bing generally favors Microsoft content more frequently—and far more prominently—than Google favors its own content.
  • Google references own content in its first results position when no other engine does in just 6.7% of queries; Bing does so over twice as often (14.3%).

The results suggest that this so-called bias is an efficient business practice, as economists have long understood, and consistent with competition rather than the foreclosure of competition. One necessary condition of the anticompetitive theories of own-content bias raised by Google’s rivals is that the bias must be sufficient in magnitude to exclude rival search engines from achieving efficient scale. A corollary of this condition is that the bias must actually be directed toward Google´s rivals. That Google displays less own-content bias than its closest rival, and that such bias is nonetheless relatively infrequent, demonstrates that this condition is not met, suggesting that intervention aimed at “debiasing” would likely harm, rather than help, consumers.

4 responses to My New Empirical Study on Defining and Measuring Search Bias

  1. 

    One thing that came to my mind was that surely you cannot compare Bing and Google in empirical evidence as they are at different stages of maturity. Bing still in my eyes, is in its infancy being released in 2009 whereas Google is rather more matured in that it started running in 2004. Thus, albeit tenuous, I would have thought that they would have different motives? So Google may have had a need to bias their content whilst Bing need to do widen the number of searches available.

  2. 

    I am the CEO of Foundem, one of the companies at the centre of the European Commission’s formal antitrust investigation into Google.

    In October 2009, we launched SearchNeutrality.org and defined search neutrality as the principle that search engine results should be comprehensive, impartial, and based solely on relevance. Clearly, no two search engines will produce the same search results; nor should they. But any genuine pursuit of the most relevant results must, by definition, preclude any form of arbitrary discrimination. The problem for Google is that its Universal Search mechanism and its increasingly heavy-handed penalty algorithms are both clear examples of arbitrary discrimination.

    By shoehorning prominent links to Google’s own services into its users’ search results, Universal Search is transforming Google’s ostensibly neutral search engine into an immensely powerful marketing channel for Google’s secondary services.

    Crucially, Google uses different ranking algorithms and relevance signals to place its own services than it uses to place everyone else’s, giving Google absolute control over how aggressively and comprehensively it commandeers any given vertical. Foundem’s study of the UK and US price comparison markets, for example, found that Google consistently exploits this control to place its own price comparison service at or near the top of its search results for virtually all product- and price-comparison-related searches.

    At the recent US Senate Antitrust hearing into Google, Senator Lee referred extensively to this Foundem study during his questioning of Google Chairman Eric Schmidt. We have produced a video highlighting some of the more important inaccuracies and inconsistencies of Mr Schmidt’s testimony. Anyone who would like to understand more about the subtext of the hearing or the ongoing antitrust investigations on both sides of the Atlantic might find this video interesting: http://www.searchneutrality.org/foundem/highlights-google-senate-antitrust-hearing

    Shivaun Raff
    CEO and Co-Founder of Foundem and SearchNeutrality.org (a Foundem initiative)

Trackbacks and Pingbacks:

  1. Investigating Search Bias: Measuring Edelman & Lockwood’s Failure to Measure Bias in Search « Truth on the Market - November 8, 2011

    […] week I linked to my new study on “search bias.”  At the time I noted I would have a few blog posts in the coming days […]

  2. My New Empirical Study on Defining and Measuring Search Bias - November 3, 2011

    […] Joshua Wright on November 3, 2011 · 0 comments [Cross posted at Truthonthemarket.com] Tomorrow is the deadline for Eric Schmidt to send his replies to the […]