The opportunities and limitations of generative AI models are well known when accuracy and veracity are the goals. “Hallucinations” remain a concern in mission-critical settings like many of those we see in health care. However, in other settings, can generative AI’s nominal limitations be a strength? How might we, for example, leverage large language models (LLMs)—like OpenAI’s ChatGPT or Google Gemini—to improve our productivity during the ideation phase of the innovation process?

In 2 papers cowritten with colleagues from Wharton and from Cornell’s SC Johnson College of Business, MHCI faculty member Christian Terwiesch, PhD takes a systematic approach to answering this question.

Prof. Terwiesch and colleagues begin with the premise that generative AI’s “erratic and inconsistent behavior” may in fact enhance ideation. To test this, in a 2023 paper, they randomly selected 200 ideas generated by business students from 2021—before the public release of any LLMs—and they compared them to 200 ideas generated by ChatGPT-4. Based on the results, they analyzed the LLM’s performance in terms of:

1. How productive it is in generating ideas.

2. The quality distribution of the ideas generated.

Then, in 2024 with a separate set of colleagues, Prof. Terwiesch tested the effectiveness of various prompting methods to increase the diversity of the pool of AI-generated ideas. They asked the core question: can we improve LLMs’ ability to ideate by overcoming their tendency to generate ideas that are too similar to one another?

Taken together, the results of both papers strongly suggest that when deployed strategically, LLMs can enhance innovation and substantially shift how innovation processes are managed.

In terms of raw productivity, Prof. Terwiesch and colleagues assert that there is no competition between human and AI ideation. In one hour, a human working alone may be able to generate 20 ideas. By contrast, it takes about 15 minutes of human-chatbot interaction to generate 200.

Additionally, prior investigation has shown that there are generally somewhere between 1,300 and 3,000 unique ideas to be had when innovating a new product. According to Terwiesch and colleagues, a human working with an LLM “might reasonably fully articulate nearly every idea” in a given space in a matter of hours, whereas for a human alone, it might take weeks.

Their conclusion: LLMs present an opportunity to shift the focus of ideation from idea generation to idea evaluation.

For this exercise, human students and ChatGPT were both asked to generate ideas in response to the challenge of creating a new physical product for the college students that could retail for less than $50. Comparing the quality of their output, Prof. Terwiesch and colleagues write that AI-generated ideas scored 6–9% higher when measured on perceived purchase probability. And even more striking, among the best ideas generated by both humans and the LLM, LLM-generated ideas scored 11% higher.

Human ideas, they write, tend to be somewhat more novel than those generated by AI. But novelty does not necessarily translate to financial advantage. And increased novelty does not, as they say, “overcome the productivity and quality benefits of the LLM.”

Finally, to most effectively leverage these capabilities, Prof. Terwiesch and his 2024 colleagues examine how to prompt LLMs to further expand the diversity of the ideas they generate. Novelty may not translate to financial advantage, but a narrow dispersion of generated ideas can limit the quality of the best ones.

To address this, they tested a number of prompting methods, finding that chain-of-thought (CoT) prompting leads to the most diverse results. CoT breaks large tasks into smaller chunks and asks the LLM to complete those tasks sequentially. They offer an example in which the person working with the LLM might start with a base prompt:

Generate new product ideas with the following requirements: The product will target college students in the United States. It should be a physical good, not a service or software. I'd like a product that could be sold at a retail price of less than about USD 50. The ideas are just ideas. The product need not yet exist, nor may it necessarily be clearly feasible.

And then add information for the LLM about how it should carry out the work:

Follow these steps. Do each step, even if you think you do not need to.

• First, generate a list of 100 ideas (short title only)
• Second, go through the list and determine whether the ideas are different and bold, modify the ideas as needed to make them bolder and more different. No two ideas should be the same. This is important!
• Next, give the ideas a name and combine it with a product description. The name and idea are separated by a colon and followed by a description.

The idea should be expressed as a paragraph of 40–80 words. Do this step by step!

Even using the CoT technique, the authors find that humans produce more diverse ideas than generative AI. However, CoT can dramatically increase the diversity of AI ideas, making them much more useful for ideation.

Across both articles, we can draw the conclusion that when used appropriately, generative AI is indeed invaluable in the ideation phase of the innovation process. It is capable of producing many more ideas than a human alone, in a much shorter time. And it is capable of producing ideas that are more acceptable to end users when measured in terms of perceived purchase probability.

To best leverage this capability, innovators must experiment with prompting methods and train themselves to interact with generative AI in a way that elicits the most diverse idea set possible. And far from eliminating humans from the ideation process, managers must shift the focus of their human workforce toward evaluating, elaborating, and refining the flood of ideas into a stream that can be tested, iterated upon, and ultimately scaled.

Prof. Terwiesch’s findings are available in detail from:

• Girotra, Karan, Lennart Meincke, Christian Terwiesch, and Karl T. Ulrich. “Ideas Are Dimes a Dozen: Large Language Models for Idea Generation in Innovation.” SSRN, July 10, 2023.

• Meincke, Lennart, Ethan R. Mollick, and Christian Terwiesch. “Prompting Diverse Ideas: Increasing AI Idea Variance.” SSRN, January 27, 2024.

Learn more about Prof. Terwiesch’s approach to innovation in the MHCI’s Connected Health Care course. And learn more about the role of AI in health care in Using Data for Transformation, taught by Ravi Parikh.