Capability, Evidence and the Benchmark Problem

Over three evenings, 14–16 February 2011, the American quiz show Jeopardy! broadcast a match between its two greatest champions, Ken Jennings and Brad Rutter, and a computer system built by IBM. The system, called Watson, ran on a cluster of IBM Power 750 servers and had been developed since around 2007 by a team under David Ferrucci. Its architecture, DeepQA, parsed each clue, generated candidate answers from about 200 million pages of stored text (reported) and scored the evidence for each with hundreds of statistical and rule-based methods before combining the scores into a confidence estimate. Watson won decisively. Its odd mistakes, notably answering "Toronto" in a category about US cities, only made the coverage friendlier.

IBM moved quickly to turn the victory into a business. In September 2011 it signed its first commercial Watson deal, with the health insurer WellPoint, and it went on to announce projects with Memorial Sloan Kettering and MD Anderson, two of the leading US cancer centres. In January 2014 the chief executive, Ginni Rometty, announced the IBM Watson Group with a reported $1 billion commitment and about 2,000 staff. On 13 April 2015 IBM launched Watson Health, and over the next year spent more than $4 billion buying health-data companies to feed it, including Merge Healthcare (~$1 billion) and Truven Health Analytics (~$2.6 billion). "Cognitive Solutions" became a reporting segment in 2016.

The evidence never caught up with the marketing. MD Anderson put its Oncology Expert Advisor project on hold in late 2016 after about $62 million of spending, and a University of Texas audit followed in 2017. In July 2018 STAT reported internal documents showing that Watson for Oncology had been trained on a small number of synthetic cases rather than real patients, and had made "unsafe and incorrect" treatment recommendations in testing (no patient harm was reported). IBM's revenue fell for 22 consecutive quarters to late 2017 (reported). On 21 January 2022 IBM agreed to sell the Watson Health data and analytics assets to Francisco Partners for an undisclosed price, reported at about $1 billion, roughly a quarter of what it had spent on acquisitions.