I recently chatted to a former PhD student, Saad Mahamood, who works for trivago, which is a travel company that uses AI. Saad told me about some of his interaction with tech vendors trying to push LLMs. This brought back some memories (not good ones) about IBM Watson.

IBM Watson and Jeopardy!

IBM Watson was a question answering system developed around 2010, which was designed to compete on the TV quiz show Jeopardy! (Wikipedia entry). In 2011, the Watson system beat top human players at the game. At the time, this gathered a lot of media attention, as it showed a computer could beat top humans at a task which required broad knowledge of the world.

The system was very impressive for the time, a striking achievement! IBM then tried to commercialise Watson, under the tag “cognitive computing”; I think it spend billions on this. At the time, Arria had some links with IBM, so I visited the Watson tech team a few times, attended some workshops and conference, and also went to a few commercial presentations.

The latter were somewhat bizarre, because they tended to pitch “cognitive computing” as the wave of the future, with slick videos (example) which pushed the message that “AI and cognitive computing are the future and will change everything” and (for companies) “join the AI revolution or you will get crushed”. But these videos and presentations were a bit light on actual success stories. One project which was mentioned in almost all of the ones I saw was a system which would help doctors treat cancer. However, this project was not successful. I recommend that anyone who is interested read Strickland’s excellent article in IEEE Spectrum, which includes the following paragraphs in its introduction

Within the comfortable confines of the dome, Watson never failed to impress: Its memory banks held knowledge of every rare disease, and its processors weren’t susceptible to the kind of cognitive bias that can throw off doctors. It could crack a tough case in mere seconds. If Watson could bring that instant expertise to hospitals and clinics all around the world, it seemed possible that the AI could reduce diagnosis errors, optimize treatments, and even alleviate doctor shortages—not by replacing doctors but by helping them do their jobs faster and better.

Outside of corporate headquarters, however, IBM has discovered that its powerful technology is no match for the messy reality of today’s health care system. And in trying to apply Watson to cancer treatment, one of medicine’s biggest challenges, IBM encountered a fundamental mismatch between the way machines learn and the way doctors work

“Dome” above refers to a room that IBM used for Watson demos. I once saw a presentation in the dome, aimed at oil/gas executive (not healthcare), and I remember one the attendees (from a major energy company) saying that this was all very impressive and exciting, but he didnt understand how Watson could help him run his oil platforms, refineries, etc more efficiently and profitably. No one from IBM responded to this comment.

The Watson system is now pretty much ancient history; the name “Watson” still exists, but it is used as a generic marketing term for IBM’s AI offerings, it doesnt refer to the QA technology developed in the above project.

Are there lessons for LLMs?

The above is history, but I bring it up because much of what I see in the LLM world resonates with things I saw in Watson many years ago. I should say that I think LLM technology is much more exciting and powerful than Watson! However, when I look at how it is being pushed, especially commercially, what I see reminds me of the above. Lots of visionaries and gurus saying “AI and LLMS change everything”, “join the revolution or get crushed”, etc. Also lots of very impressive demos, but less in the way of solid commercial success stories (I mean successful and useful products which have a real impact on large numbers of people, not success in attracting venture capital).

Echoing the Spectrum article mentioned above, I also see lots of demos in artificial contexts, which ignore real-world issues and barriers. Indeed, I highly recommend that anyone interested in real-world usage of LLMs read Strickland’s article (its only 6 pages of content), and think about how this could relate to what is happening in 2023 in the LLM world.

A common theme in the Watson and LLM stories is that big tech companies (IBM for Watson; Google, Microsoft, OpenAI, etc for LLMs) are doing their best to make money from the technology they have developed; this is what we expect companies to do in a capitalist society. And from a sales/marketing perspective, in both cases this leads to pitching their technologies as “the future” in a generic sense, as well as pitching specific solutions to real client needs.

Final thoughts

The Watson story is depressing; very impressive technology, backed by billions of investment from a leading tech company, eventually leading to disappointment. Hopefully LLMs will have a more positive outcome, but I think it is important to keep past experiences in Watson in mind…