New research titled “Do Androids Laugh at Electric Sheep?” has shown that AI is severely lacking when it comes to writing and understanding comedy.
At a time when the Writer’s Guild and the SAG/AFTRA actor’s unions are on a historic strike over issues that, among other things, specifically includes the role of AI, a career comedy writer/producer, and a cartoon voice actor weighed in on whether or not they think being funny is something AI will ever master, and muse openly about the consequences for performers and content creators if it does.
Latest Research Shows AI Doesn’t Get the Joke…Yet
In the recently published study, researchers from Cornell University set out to determine if current AI models could write comedy as well as humans and to see if AI could figure out why something was considered funny. Another recent study covered by The Debrief revealed that a properly trained AI was just as creative as its human counterparts, so the idea that they might actually grasp comedy was not all that far-fetched.
To make such a determination, the research team collected 14 years’ worth of Cartoon Caption Contest entries sent to the New Yorker. The contest involves presenting a single-frame cartoon image and then asking the reader to write a funny caption. In all, over 700 cartoons were selected.
According to a press release announcing the study and its results, each contest that was loaded into the AI included a caption-less cartoon, that week’s entries, and the three finalists selected by New Yorker editors. Some contests also included what the researchers termed “crowd quality estimates for each submission.”
With the images and captions loaded, two distinct AI models were asked to select the finalist caption for a given cartoon from among “distractors,” meaning captions that were finalists but for other contests and the actual correct answer. The AI also had to provide a “quality ranking,” differentiating a finalist caption from a non-finalist. Finally, the AI models had to generate “free text,” saying how a high-quality caption relates to the cartoon.
“The way people challenge AI models for understanding is to build tests for them – multiple-choice tests or other evaluations with an accuracy score,” said Jack Hessel, Ph.D., a research scientist at the Allen Institute for AI (AI2) and the lead author on the study. “And if a model eventually surpasses whatever humans get at this test, you think, ‘OK, does this mean it truly understands?’”
When researchers looked at the results, they found that humans were still well ahead of AI when it comes to writing, understanding, and explaining what is funny.
The best performance by an AI in matching a caption to a cartoon was a paltry 62%. For comparison, humans in the same setting were 94% accurate. A comparison of the explanations for why something was funny showed that the answers offered by humans were determined to be more accurate than their AI counterparts by a score of 2 to 1.
“There are datasets of photos from Flickr with captions like, ‘This is my dog,’” Hessel said of the disproportionate results compared to AI’s success in other areas. “The interesting thing about the New Yorker case is that the relationships between the images and the captions are indirect, playful, and reference lots of real-world entities and norms. And so the task of ‘understanding’ the relationship between these things requires a bit more sophistication (than the AI’s had).”
What do Comedy Writers and Actors Feel About AI Failing to be Funny?
With the writing and acting unions on strike and the use of AI technology considered one of the central issues, The Debrief reached out to two (temporarily) unemployed comedy professionals to get their reactions to the study.
“So much of creating comedy depends upon understanding and having lived the human experience: there’s timing, delivery, and even just understanding why connecting two seemingly unrelated concepts might make for a funny pairing,” explained Steve Callaghan, a writer, and producer who has worked for the TV show Family Guy since its 1999 inception, while also occasionally performing in movies like Ted 2 and A Million Ways to Die in the West. “So, I’m not convinced that an AI algorithm can successfully duplicate all that just yet.”
Fred Tatasciore, a voice-over actor with over 900 credits to his name, including a recent run as Yosemite Sam for Looney Tunes, also told The Debrief he thought that timing was critical to comedy. He also echoed Callaghan’s point regarding the human experience that seemingly lies at the heart of most comedy.
“Because comedy is so subjective, I can see an AI being programmed to set a certain metric of incongruity (which is a comedy basic), but it may not be able to write jokes from a life experience at this point,” Tatasciore theorized.
Both comedy veterans were concerned about sounding overly esoteric about the concept of comedy but still remained skeptical that an AI that had no emotions or experiences could generate or even understand the humor that would resonate with a human audience.
Notably, Callaghan explained how curiosity got the best of him when Chat GPT was first made available to the public, telling The Debrief how he immediately downloaded it, and pretty much the first thing he did was ask it to write an episode of Family Guy. As the person who has written the most episodes of the show during its nearly two-and-a-half-decade run, as well as someone who has to select between jokes written by the show’s other comedy writers, Callaghan was both curious and somewhat concerned regarding how well the AI tool would perform. Fortunately, he says, his worries were quickly allayed.
“It was not good,” Callaghan said with a chuckle at the AI’s unsuccessful script attempt. “I mean, it had the main character names right, but the format was all wrong, and the story didn’t really make sense.”
When asked by The Debrief if the punch lines were funny, Callaghan laughed again before offering an interesting answer that seemed eerily consistent with the results of the recent study.
“No, they weren’t even really jokes,” he said. “(The punchlines) read more like riddles. Like it (the AI) was trying to write comedy but couldn’t really grasp it.”
Do Our Comedy Professionals Believe AI can One Day Figure out How to be Funny?
After some further discussion regarding the study and its results, both Tatasciore and Callaghan say they don’t believe it is entirely out of the question that AI could one day grasp and even create humor. In fact, Tatasciore says he can already see his pre-school-aged son growing his own sense of humor as his brain becomes more complex and his life experience expands, much like AI models seem to get smarter over time.
“Even now, I see my 4-year-old son developing his comedic sense of incongruity with superhero monikers, starting with “Mega,” the voice of Marvel’s The Hulk said. “And he will go down the line with “Super Mega-Grandma!”, “Super Mega-Poop!” and even “Super Mega-Grandma Poop!”
“Poop is always funny, especially at 4,” he added.
Callaghan agreed, at least about the growth potential of AI, also noting how AI software has already improved in the second and third generations. However, when asked by The Debrief if he thought an AI comedy revolution would help or hurt his industry, something producers and talent are both grappling with in current WGA and SAG/AFTRA negotiations, the fear nearly all folks who write or perform for a living finally seemed to sneak out.
“I’m sure machine learning will get better over time, so AI’s ability to generate comedy may improve somewhat, but I don’t see it ever fully replacing a human comedy writer,” he said. “At least that’s the mantra I repeat to myself each night as I lie awake, sweatily panicking I’m not as funny as a robot.”
When asked the same question, Tatasciore echoed a similarly grave concern.
“AI is getting better with its timing and vocal nuances, but as yet, maybe not inspired as a human can do. But who knows? Give it time!” he said before adding, “Yeah… we’re screwed.”
Mega poop.
Christopher Plain is a Science Fiction and Fantasy novelist and Head Science Writer at The Debrief. Follow and connect with him on X, learn about his books at plainfiction.com, or email him directly at christopher@thedebrief.org.