Update 27-Apr-23: An excellent example of a good evaluation of GPT is “Large language models effectively leverage document-level context for literary translation, but critical errors persist” (https://arxiv.org/abs/2304.03245). I ecnourage anyone who is interested in rigorous evaluations of GPT to read this paper!

Occasionally people ask for my advice on evaluating chatGPT (or GPT4). I love getting such questions, because they are much more constructive than, say, debating whether chatGPT is “Artificial general intelligence” (AGI) or a threat to humanity. My cynical view is that much (not all) of this debate is driven by people whose “business model” requires getting the attention of lots of people (politicians, journalists, pundits, think tanks, etc). Such people want to capitalise on chatGPT’s prominence and visibility. Since they cannot comment technically and have zero interest in careful scientific experiments, they instead start pontificating on whether AI is a threat to humanity.

To me, the most important question about GPT and LLM technology more generally is what it can and cannot do. We need to understand this in order to know when and how to use the technology, and indeed where it needs to be improved. Exploring this issue requires careful high-quality scientific experiments (and I am sick of the prominence given to press releases, cherry picked examples, dubious experiments, etc). If anyone wants to do such experiments, I am happy to help and give advice! Below I give an example and some pointers on what not to do.

Example: chatGPT in health chatbots

One of my PhD students, Simone Balloccu, is interested in using chatGPT in health chatbots. I dont want to give details here since this is ongoing work, but you can get a general idea of the sort of thing Simone is interested from Wu, Balloccu et al 2022 and Balloccu and Reiter 2022.

Anyways, from an evaluation perspective, what Simone has done is

1. Get a set of novel inputs (essentially statements and queries in a specific health domain); Simone gets these from crowdworkers (Mechanical Turk and Prolific).
2. Run these inputs through chatGPT to get responses to the above inputs.
3. Ask domain experts to evaluate the quality and appropriateness of the generated feedback.

As mentioned, this is ongoing work, but the preliminary results do suggest strengths and weaknesses. I wont go into this here, except to say that some of what Simone is finding supports my earlier observations that health-related LLM output can be inappropriate even if it is accurate. Incidentally, I myself would not use chatGPT for health information. If I want online information about a health issue, I will use a Google search and click on a website I trust, such as https://www.nhs.uk/.

This approach is not limited to health. I have an MSc student who is doing something similar to investigate chatGPT in a software development context, and he is also finding out interesting things about strengths and weaknesses. Hopefully I’ll have more MSc students working in this area over the summer. The approach probably wont work everywhere, but I think it does work in many contexts.

An even better experimental approach is to deploy the system in real usage and study its impact. Ive not seen any such studies of chatGPT to date, but we described one such study (using a different technology) in Knoll et al 2022.

Advice: Do not evaluate chatGPT on Internet data

The above approach involves soliciting new input test data. Of course most people who claim to evaluate chatGPT dont do this, instead they get test data from the internet. Some people even use test sets where (input, output) pairs are available on the internet, such as leaderboard data sets. This is a **REALLY** bad idea. Since chatGPT was trained on a big chunk of the Internet, “evaluating” it on test data which is available on the net amounts to testing chatGPT on its training data, which violates one of the fundamental principles of machine learning and means the results are worthless.

I’m also wary of testing chatGPT on inputs from the public internet even if a formal (input, output) dataset is not available, because in many cases corresponding outputs do exist for the inputs, even if there isnt a formal leaderboard training data. For example, if we ask chatGPT to describe a sports match and give it data from a real game, chatGPT could simply find a story written by a human sportswriter about this match and regurgitate it. And if we ask chatGPT to pass an exam, we need to ensure that the exam questions (perhaps in a different form) are not available on the Internet.

Having said this, it is possible to do a proper experiment using Internet data (a good example is Vilar et al 2022). However this is not easy and requires a lot of careful attention to above issues. For most people I think the safest approach is to use test data which is not on the internet. This could be crowdsourced data (which Simone used), or it could be data which was never publicly released because of commercial or data-protection issues.

Advice: Evaluate chatGPT outputs with human domain experts

The above approach also asks human domain experts to evaluate the quality of chatGPT’s texts. Again this is unusual, most NLP researchers use metrics or ask random Mechanical Turkers to evaluate quality. I appreciate that using metrics is a lot cheaper and quicker than using domain experts, but the results from domain experts are a lot more meaningful and trustworthy! Especially if part of the investigation is identifying texts which are inappropriate in more subtle ways, such as making users unnecessarily scared or depressed. I have discussed this in previous blogs (example).

Advice: Avoid commercial conflict-of-interest

It is better if the experiments are done by researchers who are not connected to the company which is developing and selling this software. My above-mentioned MSc student made this point strongly to me a few days ago. After having read a variety of papers, he is now very skeptical about experiments done by researchers who have a commercial link to the product they are evaluating. Researchers are expected to declare commercial conflict-of-interestes in medical research, perhaps this should be required in AI as well. I certainly am very wary when reading about experiments where the authors have a clear commercial interest in a positive outcome.

Caveat: Reproducibility

Unfortunately, experiments done with chatGPT are generally *not* reproducible, because OpenAI is continually updating chatGPT. For example, if Simone reruns his inputs through chatGPT a month later, he often gets different responses, which means the evaluation by domain experts may be different.

This is a real pain from the perspective of doing careful science! But I think that we still need to make the effort, and my personal view is that even if the detailed results of the evaluation change when an experiment is rerun, in most cases high-level insights about strengths and weaknesses will persist (but this is based on a few anecdotal examples, I have not properly tested this hypothesis).

To put this another way, even taking this issue into account, any careful scientific evaluation of chatGPT is probably going to be more meaningful than the alternative, which is low-quality experiments, cherry-picked examples, etc.

Final thoughts

In order to use LLM technology to help people, we need to do proper experiments to understand the strengths and weaknesses of the technology. So far I have seen very few such experiments, which is a real shame. High-quality experiments require a lot more work than “experiments” based on applying weak metrics to published leaderboard data, finding impressive cherry-picked examples, etc, but they are the only way to really explore what LLM technology can and cannot do.

In short, I strongly encourage people who want to understand how to use LLM technology to do high-quality experiments, using novel test data as input and domain experts to evaluate output quality. And if you think I can help, please ask!