Using AI generative in the foresight practice

At its best, foresight brings together diverse voices and perspectives to imagine ‘what if’. So what if we included artificial intelligence (AI) alongside these voices?

Generative’ AI — tools that can generate text, images, videos, music and other media in response to a user’s prompts — have proven to be impressive performers at many tasks, including designing strategies and research, summarising evidence and looking for knowledge gaps, and conducting analyses. All key skills for strategic foresight practitioners.

Recently, my team at British Red Cross rapidly sketched scenarios for how the UK’s cost of living crisis might evolve over the coming years — so rapidly that we didn’t have time to bring actual humans together for the necessary workshops. But this gave us the opportunity to explore how AI could become a part of our strategic foresight toolkit.

The first thing I tried was asking three generative AI tools — ChatGPT, Bing AI and Perplexity — to come up with some scenarios for the cost of living crisis. The results were deeply uninteresting — Bing just gave options where the crisis improves, stabilities or worsens, while Perplexity only looked at individual drivers of change in isolation. So far, so unimaginative.

Then it struck me that AI could help at many stages in IARAN’s approach to strategic foresight:

• PESTLE analysis

• Identifying the drivers of change and actors in the system you’re studying

• Judging the impact and uncertainty of the drivers of change

• For each driver of change: researching their history, and analysing indicators and trends

• Coming up with hypotheses for how drivers of change might evolve in future

• Writing scenarios from a selection of hypotheses.

PESTLE analysis and identifying drivers of change

IARAN’s approach starts by understanding the dynamics of the system you want to study (the UK’s cost of living crisis, in our case) through its Political, Economic, Social, Technological, Legal and Environmental (PESTLE) dimensions, and then mapping the ‘architecture’ of your specific system, the ecosystem in which it is embedded, and the global system.

My team has monitored and analysed the cost of living crisis over the last year and a half, so mapping the architecture was a fairly quickly job — but it was helpful to have ChatGPT on hand so I could ask it questions like, “What are the key drivers of living standards?” or, “What are the drivers of change for the cost of living?” and see if we’d missed anything obvious.

Even simple questions such as “What causes prices to rise?” can give helpful context if you are less familiar with a system (though I’d recommend using a generative AI tool that cites its sources, such as Bing or Perplexity and always having human expertise in the loop).

Impact and uncertainty

The next stage involves judging the impact and uncertainty of each driver of change.

For this, I wanted to include a diversity of AI-generated views so I called on five generative AI chatbots: ChatGPT, Bing, Perplexity, Google Bard, and Claude.

I ask each of the five tools the same question: “Rank the following drivers of change based on their uncertainty and their impact upon of UK’s cost of living crisis: [list of drivers].”

The AI chatbots had a good go at scoring the drivers of change, but they gave incomplete and inconsistent results. After tweaking my prompt, the chatbots provided helpful tables of scores (here’s an example from ChatGPT) that I copied into Excel for further analysis.

Each tool varied in its ability to answer the question. Bard, ChatGPT and Perplexity gave impact and uncertainty scores for each of the 42 drivers of change. Claude missed one of the drivers of change and gave multiple scores for two other drivers, strangely. Bing didn’t provide scores for four of the 42 drivers. The five AIs ended up disagreeing with one another in how they categorised impact and uncertainty across the drivers of change, according to a measure of inter-rater reliability.

But what was the quality of answers like? Surprisingly good. There were a handful of cases where an AI made an odd choice — for example, Perplexity categorised unemployment as having a low impact on the cost of living crisis, which I (and the other AIs) disagree with. On the whole, I was impressed — especially by Claude.

Again, subject-matter knowledge (in humans) is key here: we must always check for confidently delivered nonsense (from machines as well as humans).

Now I needed to find a way to aggregate the impact and uncertainty scores from each AI into a single impact-uncertainty matrix. For humans, we might use something like dot voting or forced ranking to do this. For the AIs’ scores, I tried a few different approaches — including calculating arithmetic and geometric means, modes, and the number of times a driver was scored as ‘high’ — eventually settling on summing the scores from the AIs then dividing them into terciles (i.e. ‘low’, ‘medium’, and ‘high’). From here, it was a simple case of mapping the drivers onto an impact-uncertainty matrix:

Normally at this point you would write driver files for the critical scenario drivers (those which are high-impact and high-uncertainty). Due to time constraints, we had to skip this stage and went straight to hypothesising how the drivers could change in the future.

Hypotheses for how drivers of change might evolve

Now we need to come up with a set of mutually exclusive hypotheses for how each high-impact, high-uncertainty driver could change over the coming years.

I asked Claude to “Generate hypotheses for how these factors could change over the next 5 years: [list of critical scenario drivers]” and received this response:

These were not terribly useful hypotheses for the purposes of foresight, so I ended up writing my own.

Writing scenarios

I couldn’t figure out a way to get the generative AI tools to create hypothesis chains to form the basis of scenarios — so I manually created ours:

But a way around this (which I only thought of later) could be to just put every hypothesis into a generative AI and see what it comes up with:

From this prompt, ChatGPT outlined four scenarios, which it also named: “Progressive Climate Change and Economic Recovery”; “The Dire Straits”; “The ‘Green’ Dystopia”; and “The Technological Breakthrough.”

Here’s the ‘Green’ Dystopia scenario in full:

“The cost of living crisis in the UK has evolved in a surprising manner. Innovation in food production and distribution has raised environmental and ethical concerns, causing a divergence in food prices based on these factors. Similarly, while energy transition has diversified and created opportunities for cooperation and competition, it has also introduced new risks, leading to fluctuating energy prices. The diversification of energy sources has improved energy security but has led to new trade-offs. Despite the emergence of new jobs in the green economy, the transition has led to job losses in other sectors, increasing overall unemployment. The affordability crisis has been somewhat alleviated by the adoption of low-cost solutions, but behavioral changes and policy support haven’t been able to keep up with the pace, leading to ongoing struggles for many households.”

Reflections, tips, and ideas

This current generation of generative AI chatbots proved to be helpful assistants, particularly for checking my own biases during the processes of rapidly developing cost of living scenarios. Though it’s always worth remembering that AI reflects the biases and worldviews of the data that goes into training them — which generally excludes the most marginalised and minoritised people — so proceed with caution if bringing AI into your foresight process.

I found these tools most useful during the earlier stages: identifying and classifying the drivers of change. I was less impressed with the more ‘creative’ stages of generating scenarios, but that could just be a signal that I needed to use better prompts. It’s not that generative AI can’t do creative work, and it will only improve. As Ethan Mollick states, currently available AI tools are the worst you’ll ever use.

There are still a few more realms within strategic foresight that I want to explore with AI assistants:

• Horizon scanning, particularly for weak signals of change (here’s my first attempt at cajoling ChatGPT to identify weak signals).

• Using ChatGPT’s new ‘code interpreter’ feature to help analyse drivers of change. Ethan Mollick has written a great overview of this powerful feature, which could help bring more quantitative power into strategic foresight.

• Plug a generative AI into a curated knowledge base containing research and analysis of drivers of change — to see whether it can identify signals that we humans might miss, or if it can help us imagine previously unimaginable futures.

*This article was originally first published by its author on Medium

References

1. "Insight and Improvement at British Red Cross | Medium" - [URL: "https://medium.com/insight-and-improvement-at-british-red-cross/humans-and-machines-imagining-futures-together-68767ea8c673"]

2. "ChatGPT" - [URL: "https://chat.openai.com/"]

3. "Bing AI" - [URL: "https://bing.com/new"]

4. "Perplexity" - [URL: "https://www.perplexity.ai/"]

5. "IARAN’s approach to strategic foresight" - [URL: "https://www.iaran.org/strategic-planning"]

6. "Google Bard" - [URL: "https://bard.google.com/"]

7. "Claude" - [URL: "https://claude.ai/"]

8. Saal, F.E., Downey, R.G., & Lahey, M.A. (1980). Rating the ratings: Assessing the psychometric quality of rating data. Psychological Bulletin, 88(2), 413. doi:10.1037/0033-2909.88.2.413.

9. "How generative AI could disrupt creative work" - [URL: "https://hbr.org/2023/04/how-generative-ai-could-disrupt-creative-work"]

10. "Ethan Mollick on Twitter" - [URL: "https://twitter.com/emollick/status/1678226869949169668"]

11. "ChatGPT’s 'code interpreter' feature" - [URL: "https://openai.com/blog/chatgpt-plugins#code-interpreter"]

12. "What AI can do with a toolbox" - [URL: "https://www.oneusefulthing.org/p/what-ai-can-do-with-a-toolbox-getting"]