A few weeks ago I gave a seminar within Aberdeen CS on non-academic “impact” of academic research, focusing on my own experiences. It triggered a lot of thought-provoking discussion, so I thought I’d write a follow up blog.

What is impact?

Most research funding comes from governments, and most government research funders want to see that research (over some time span) produces benefits for citizens and taxpayers. Of course, not all research will have real-world impact, and if there is impact, it may take years or even decades to appear! But overall, there is an expectation that citizens will see benefits from government research spending. This is especially true in STEM fields such as Computer Science, and is one reason why CS gets a lot more research funding than Philosophy (for example). In my experience, most CS academic researchers also want their research to be useful in the “real world.”

In the UK, the government formally assesses research impact as part of the Research Excellent Framework (REF) exercise. REF defines impact as “an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia.” This is not a perfect definition, but I think its as good as 1-sentence definition I’ve seen. Its also important to UK academics because its used by REF.

Impact of Computing Science Research

REF asks for concrete “case studies” of impact, and in computing most of the ones I’ve seen fall into one of the following categories:

• Technology: algorithms, models, open-source software, etc which are used by people outside of academia. This can be government, charity, and private individuals as well as companies.
• Spinout companies: companies founded by academics based on their research. As well as technology (above), companies also benefit from insights about requirements, methodology, human factors, and other “non-algorithmic” factors,
• Policy and standards: cybersecurity and AI policies for governments; setting standards for internet; etc
• Public engagement: giving talks, creating websites, and otherwise letting the public know about your research. Getting people (especially children) excited about science benefits society!

There are probably other types of impact, above list is not exhaustive.

Impact also comes at different scales. At one end of the spectrum, the foundational deep learning work by LeCun, Bengio, and others has led to technology which has benefitted millions (billions?). At the other end of the spectrum, a talk to ten children at a local science centre might get a few of them excited about science. Most impact falls between these two extremes.

Barrier: Work on useful problems

So what are the barriers to achieving impact? The first one is that it really helps to work on problems that real-world users care about, and address the concerns they have. A colleague recently commented to me that he sees a lot of recent work on NLG which tackles made-up problems that no one outside of academia cares about. Of course, such work may lead to insights that benefit real-world problems. However, I think impact is more likely if researchers focus on real-world problems from the beginning.

Similarly, its clear that accuracy and hallucinations are of paramount importance to most NLG users, but many NLG researchers essentially ignore accuracy in their work. To take a concrete example, I’ve seen a number of papers on generating sports stories, which is a real-world use case, but 90% of the academic papers do not measure accuracy rigorously; instead they use metrics which do a poor job at detecting simple accuracy errors and are completely useless at detecting more complex errors. If you ignore accuracy issues when developing a model, its unlikely to be of use to anyone in the real world.

Barrier: Commitment to impact

One of the junior faculty at Aberdeen CS commented to me that he would love to see his research used in real world contexts, but he couldnt spend much effort on this. His focus has on publishing lots of research papers, because this is the way to advance his career. I completely understand his situation, but in all honesty this makes it unlikely that his work will break out of academia. In most cases, achieving significant real-world impact requires years of effort and commitment.

Of course, one way to advance an academic career is to attract research funding, and there are plenty of dedicated funding streams for creating real-world impact from research. In all honesty I’ve become a bit cynical about such programmes, though. I’ve seen a lot of cases where the money went to individuals who told a good story but didn’t really care about impact. I’ve also seen cases where applicants who genuinely were seeking impact applied for such funding when they would have been much better off trying to attract investors.

My experiences

Above is pretty abstract, so I thought I’d talk about some of my own successes and failures at impact.

My biggest success in this area is probably Arria, which started as a spinout from the NLG research at Aberdeen Uni. Oversimplifying a bit, Arria generates two kinds of impact

• Products: Arria’s NLG products help people around the world, for example its business intelligence tools help people understand and take action on business data.
• Employment: According to LinkedIn, Arria employs 145 people (its actually more than this, but I cant give details here). 145 jobs, most of them high-skilled and well-paid, is a definite contribution to society.

A more moderate success is AIber (previously called MIME). This is a spinout company from a project I was involved in ten years ago, which focused on tools to help first aiders and first responders in remote areas support and help people with medical problems while waiting for an ambulance. The company took the technology, originally developed to support remote Scottish villages, and used it build tools for aircraft and ships, which have similar problems.

Aiber is more representative of spinouts that Arria. It employs 7 people according to LinkedIn, and its products are still getting established in the market (in part because Covid shut down air travel for much of the past two years). So at the moment its impact is more modest, but of course this will change if its products take off and are widely used by the aviation and maritime industries.

Finally, I thought I’d mention a project which failed to achieve much impact, How Was School Today. This project developed a tool to help non-speaking children write a story about their school today for their parents. It attracted a lot of media attention (including a piece on national BBC news), which got the funding agency very excited. However, in the long run the technology was not used, in part because the children we worked with were very different, which made it hard to develop something which many children could use. In other words, we could develop great tools for individuals, but this is not cost-effective in the real world.

Final thoughts

Of course, some people have more interest in impact than others! If you are keen on seeing your research help people, I suggest you try to stay grounded in what problems people outside of academia care about. You also need to appreciate that getting your ideas into real-world usage is probably going to require a lot of effort.

From a larger perspective, society (and most research funders) want research to provide real-world benefits. Of course this isn’t always possible, not least because the nature of research is that most things we try are not very successful. But regardless, I think it is important for academics to try to achieve real-world “impact” when possible. From this perspective I am disappointed that the incentives faced by young researchers often seem to push them towards churning out academic papers instead of getting their ideas into real-world usage.