An experiment revealed the ups and downs of artificial intelligence (AI) in creative writing after participants with less inherent skills showed increased creativity at the expense of low variety among similar and predictable short stories.

Published in Science Advances, the study was conducted as a stepping stone to investigate how large language models (LLM) and generative AI affect human activities.

“While there is huge potential for this technology to have big impacts in media and creativity more generally, it will be important that AI is actually being evaluated rigorously rather than just implemented widely under the assumption that it will have positive outcomes,” stated University of Exeter professor Oliver Hauser, who worked on the experiment with University College London co-author Dr. Anil Doshi, in an email to TechCrunch.

Writing stories with the help of AI

The experimental setup involved 293 participants who were asked to list ten words that were completely unrelated to each other as part of a divergent association task (DAT) designed to initially determine their inherent creativity.

They were then assigned to write an eight-sentence short story on one of three topics, including an adventure in the jungle, on an open sea, or on another planet. Afterward, one-third of the participants were granted a single three-sentence story starter using ChatGPT, while the others were divided into those who could request up to five AI help and those who did not receive any assistance.

The short stories were judged by another 600 participants who had no idea about the use of AI-generated ideas. These individuals rated the stories on a nine-point scale based on novelty, which served as the proxy for originality, usefulness in terms of likelihood of publishing, and emotional enjoyment.

AI boosts individual creativity—for some

The findings indicated that participants who scored low in DAT, suggesting a lack of inherent creativity, showed improvements when given the chance to use AI. Specifically, those who had access to five AI requests saw an average of 8.1% increase in novelty and 9% in usefulness when compared to those who wrote on their own.

Likewise, it was found that short stories from the groups given with AI-generated starters were more likely to include plot twists, making them more enjoyable. “The effect sizes are not very large, but they were statistically significant,” noted Dr. Doshi.

However, the same thing was not observed for the more naturally creative participants. According to the results, AI had little to no effect on the stories written by those who achieved high creativity metrics in DAT.

“We do not find that the most inherently creative people’s stories are being ‘supercharged’ from AI ideas, this group of people is highly creative with and without the use of AI,” Dr. Doshi explained.

Battling predictability

Despite these seemingly positive results, Prof. Hauser stressed that the experiment exposed a social dilemma.

“It may be individually beneficial for you to use AI, but as a society, if everyone used AI, we might all lose out on the diversity of unique ideas. And, arguably, for creative endeavors, we might sometimes need the ‘wild’ and ‘unusual’ ideas,” he said.

The researchers reached this conclusion after conducting their analyses using OpenAI’s application programming interface to determine the similarities among the short stories in each group.

They discovered that the use of generative AI reduced the variety of the written outputs in a group, with the data revealing that the total difference only ranged from 9% to 10%.

Dr. Doshi claimed that this might be because AI typically generates relatively predictable story ideas. He also added, “Evaluating the use of AI will be essential in making sure that we reap the benefits of this potentially transformative technology without falling prey to potential shortcomings.”

Both authors acknowledged that their experiment was limited and that future research on AI could lead to numerous paths. They also recognized that the implementation of generative AI “in the wild” has multiple differences from their controlled setting.

Nevertheless, Prof. Hauser says, “Ideally, our study helps guide both the technology and how we interact with it to ensure continued diversity of creative ideas, whether it is in writing, or art, or music.”