There was no single winner in the Bing It On dispute. Microsoft said a study it commissioned found people preferred Bing’s web results nearly two to one; a separate randomized experiment by Yale Law School authors reported that more participants preferred Google, and that the outcome shifted with query choice. The tests measured different groups under different conditions, so neither result proves that one search engine was better for everyone.
What was the Bing It On challenge?
Microsoft announced Bing It On on September 6, 2012. The public challenge let a person choose five queries, then view Bing and Google web results side by side without the search-engine branding and pick Bing, Google, or a draw. Microsoft said the comparison was limited to the web-results pane: it excluded ads, Bing Snapshot and Social Search, and Google’s Knowledge Graph. Microsoft’s announcement distinguished this public, five-query test from the commissioned study it cited in its advertising.
Microsoft described the challenge as “a fun, non-scientific online test designed to show that the quality of Bing’s web search results has surpassed Google’s,” as corporate communications vice president Frank X. Shaw put it. That description matters: a consumer-facing challenge people could take with queries of their choice was not itself the same exercise as a controlled study.
What did Microsoft claim, and what did Yale report?
Microsoft’s commissioned comparison
In its 2012 announcement, Microsoft said an independent study commissioned from Answers Research found that participants chose Bing’s web results over Google’s “nearly 2-to-1” in blind comparison tests. Microsoft Bing corporate vice president and chief marketing officer Mike Nichols said that “Relevancy of search results is the No. 1 driver of search engine preference, and independent research shows that people chose Bing’s Web search results over Google’s nearly 2-to-1.” This was Microsoft’s account of a commissioned study—not the result of everyone who used the public challenge. The announcement does not set out enough of the study’s methods to treat the figure as a universal measure of search preference.
#1 Best Overall
- Real scientific deduction: Use the companion app to survey for objects and apply logic rules to deduce Planet X's location like an actual astronomer conducting genuine cosmic investigation
- App-integrated gameplay: The free companion app randomizes object arrangements and Planet X location each game, ensuring fresh strategic challenges with procedural generation that guarantees replayability
- Strategic publishing system: Publish theories and conduct searches once you believe you've solved the puzzle, creating multiple decision points that reward careful observation and logical deduction
- Perfect for group discovery: Designed for one to four players ages thirteen and up to collaborate or compete in sixty to seventy-five minutes, making it ideal for game night with friends or family
- Trusted designer pedigree: Created by acclaimed designers Ben Rosset and Matthew O'Malley with beautiful artwork, published by Renegade Game Studios, a respected name in modern strategy board games
The Yale authors’ experiment
In “A Randomized Experiment Assessing the Accuracy of Microsoft’s ‘Bing It On’ Challenge,” Ian Ayres and co-authors report an experiment with US-based Amazon Mechanical Turk participants on Microsoft’s challenge site. Their abstract reports that 53% preferred Google results and 41% preferred Bing. It also says participants were significantly less likely to prefer Bing when assigned popular or self-selected queries rather than terms Microsoft recommended for the challenge. The authors conclude that the results “suggest that some of the claims implicit in Microsoft’s advertisements warrant legal scrutiny.” That is the authors’ assessment, not a court ruling.
The Yale paper is not a reanalysis of Microsoft’s commissioned study. The figures come from different tests, with different recruitment and query conditions; the available abstract does not establish all the details needed to compare their methods precisely. The disagreement therefore challenges how broadly the advertising claim should be read, not whether every Microsoft study was disproved.
Rank #2
- Solve Mysteries, Find Objects: Search Party is a colorful adventure and found it game for kids. Open the 3D board and follow the characters across the pop-up amusement park attractions to solve mysteries, answer questions, and more! Once you're done, you can work your way through 300 search-and-find items.
- Ages 8+: This game is intended for players 8+, so you can play with pretty much the whole fam!
- A Party For Everyone: This epic pop-up board makes this game a great gift for birthdays, housewarmings, and family game night. Play on a rainy day or for some extra fun time with the kids.
- Includes: 1 Pop-Up Board, 15 Missions (83 Cards), 1 Search & Find Guide, Instructions, 1 Mini Magnifying Glass
- Brought to you by What Do You Meme? Check out our full collection of fun stuff for families, kids, and adults.
Why did query selection matter?
Search results are responsive to the words entered. A test using terms selected or recommended for a challenge can produce a different preference pattern from one using popular queries or terms chosen by participants themselves. The Yale authors specifically report that preference for Bing was lower under popular or self-selected queries than under Microsoft-recommended terms.
That finding does not show that any one query set is inherently the correct way to evaluate a search engine. It does mean the headline result depends in part on what people searched for. A five-query challenge shaped by each participant’s interests answers a different question from a test that assigns common searches, and neither alone captures every user’s needs.
How do the reported results compare?
| Test or report | Who reported it | Participants or queries described | What was compared | Reported outcome |
|---|---|---|---|---|
| 2012 blind comparison cited at launch | Microsoft, citing a study commissioned from Answers Research | The announcement does not provide full study details | Web results; Microsoft said the comparison was blind | Microsoft reported Bing was chosen “nearly 2-to-1” over Google. Microsoft, September 6, 2012 |
| Public Bing It On challenge | Microsoft | Five queries chosen by each participant | Unbranded web-results panes; a draw was allowed; ads and specified special panels were excluded | Not the same result as the commissioned study; the challenge itself asked users to choose a winner or draw. Microsoft, September 6, 2012 |
| Randomized experiment on the challenge site | Ayres and co-authors, Yale Law School authors | US-based Amazon Mechanical Turk participants; abstract reports query conditions including popular, self-selected, and Microsoft-recommended terms | Results shown through the challenge site; the abstract reviewed does not establish every page element or methodological detail | 53% preferred Google and 41% Bing; preference for Bing was significantly lower with popular or self-selected queries than with recommended terms. Paper abstract |
| Follow-up survey reported in October 2012 | Answers Research, as reported by Microsoft | 4,706 web surveys; Microsoft said results were weighted to the US online population by age, gender, and income; Google-primary status was self-stated | Respondents’ reactions after taking the challenge, not a direct head-to-head preference tally described as the original study | Microsoft reported 64% were surprised by Bing’s web-results quality, more than half said their impression of Bing improved, and 33% of self-identified Google-primary users said they would use Bing more often. Microsoft, October 2, 2012 |
| Popular-query comparison reported in 2013 | Microsoft | Popular web queries; further participant and method details are not stated in the cited post | Pure web results without ads or the specified special panels | Microsoft reported 52% preferred Bing, 36% Google, and 12% rated them equally; it gave a margin of ±3 percentage points at 95% confidence. Microsoft, February 2013 |
What can these studies establish?
They establish that different comparison conditions produced different reported preferences. Microsoft’s campaign cited a commissioned blind comparison favoring Bing; the Yale authors’ experiment found more participants preferred Google and identified sensitivity to query selection; Microsoft later published separate survey and popular-query results favoring Bing. Each figure needs to remain attached to the test and source that produced it.
They do not establish a timeless overall winner. The Microsoft announcements do not fully describe the underlying commissioned study, and the Yale abstract does not supply its full sample size, randomization protocol, or complete statistical methods. Nor do the results cover every feature people may use in either search engine: the comparisons were framed around web results and excluded specified ads or special panels.
Quick Recap
Best Value
- A TWIST ON CLASSIC BATTLESHIP GAME: This Amazon Exclusive Battleship with Planes board game features airplanes in addition to ships for awesome strategic gameplay and epic battles
- SINK SHIPS AND CRASH PLANES: With this strategy naval combat game, fans of the Battleship game can crash enemy planes in addition to sinking ships to win
- PORTABLE GAME WITH STORAGE: This Battleship game includes 2 portable battle cases and convenient ship, plane, and peg storage. It's an ideal travel game for kids—dive into an instant game on the go
- A CHILDHOOD FAVORITE: Remember playing the Battleship strategy game as a kid? Enjoy playing this 2 player board game with a new generation
- GREAT FOR FAMILY GAME NIGHT: Looking for fun indoor games for game night? This strategy board game for kids is a great choice
Rank #4
- Sequel deduction experience: Follow the hit game The Search for Planet X with logic-based gameplay that challenges your mind to identify Lost Species through strategic reasoning and careful observation
- App integration for confirmation: Report your sightings through the free companion app to verify discoveries, creating a modern hybrid experience that blends physical and digital gameplay seamlessly
- Double-sided island maps with six species: Explore two unique terrain configurations and hunt for six different Lost Species across Indonesian and Papua New Guinean habitats, ensuring endless replayability
- Sustainable production quality: Renegade Games Studio prioritizes eco-conscious manufacturing, delivering premium components with beautiful wildlife artwork and sturdy construction for lasting game nights
- Flexible player count and accessible timing: Play solo or with up to four players in approximately 60 to 75 minutes, making it perfect for intimate gatherings or competitive strategy sessions with friends
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
