In February 2023, New York Times columnist Kevin Roose shared a bizarre dialogue he had with a chatbot in Microsoft Bing—a conversation highlighted by the program declaring Roose was unhappy in his marriage and loved it rather than his spouse. Roose’s article seemed to confirm the worst fears of artificial intelligence pessimists that the technology could quickly slip out of human control and follow its own motivations.

In November 2023, I asked my students in the first half of the American literature survey to use a chatbot to help them create an interpretation of a course text. My class described far more benign exchanges than what Roose experienced, but when I asked them to reflect critically on these programs, they found it hard not to narrate their use of the programs as a dialogue with an often obtuse conversation partner—even as we repeatedly discussed how large-language models can’t be independent contributors in the same way humans can.

The linguist Emily Bender has described this tendency as one of the dangers of “stochastic parrots.” She coined this phrase to describe how chatbots generate strings of language that appear as a considered response to a human-supplied query, but are actually patterns based largely in the collocation probability of certain words.

As someone who researches and teaches early American literature, stochastic parrots make me think of Wieland, Charles Brockden Brown’s 1798 novel about the tragedy that befalls a Pennsylvania family when they begin hearing unexplained voices in the air around them. A Gothic novel written decades before the telegraph was invented, much less the server farm, might seem an odd connection to generative AI. But Brown asks questions of agency, authenticity, and accountability that encourage us to consider how we grant such qualities to the mysterious voices that emerge from our machines.

Here’s a quick plot summary of Wieland: Clara Wieland narrates the novel as a long letter to an anonymous correspondent. She describes her family’s suffering after her brother Theodore murders his wife and children, and then seeks to murder her, all while claiming God directed him to these crimes. Prior to this violence, the Wieland family had been hearing unexplained voices that are revealed as the work of a mysterious stranger named Carwin. Carwin practices what Brown calls biloquism—he can both throw his voice and imitate the voices of others with uncanny precision. Carwin reveals to Clara near the novel’s climax that he used his ability out of a perverse desire to test the virtue and rationality of the Wielands, but pledges he did not command her brother to murder his family. Carwin later saves Clara from her brother’s assault by using biloquism to order Theodore to stop attacking his sister; this encounter restores Theodore to his senses, and he commits suicide in remorse for his actions.

Brown invented the word biloquism. In a footnote, Brown explains this term as essentially a synonym for ventriloquism, but within the story, Carwin emphasizes his ability to speak in two voices, telling Clara he can “mimic exactly the voice of another.” This definition differs from ventriloquism, which refers to the ability to speak without moving the lips, creating the illusion that the sound comes from somewhere other than the speaker. Etymologically, ventriloquism comes from Latin words for stomach and speech, while Brown coins biloquism to mean double speech. 

As the plot of the novel makes clear, the risk of biloquism lies not just in changing the perceived direction of speech, but in counterfeiting the voice of others. Carwin begins his challenge to Clara and her family with a seemingly innocuous imitation of Theodore’s wife Catherine, but later mimics Clara’s voice to convince her friend and potential suitor Henry Pleyel that Clara and Carwin are in a sexual relationship. I don’t think it’s too much of an anachronism here to say Carwin makes Clara the subject of a deepfake.

However, the novel shifts its suspense away from this impersonation toward the murders Theodore commits and the unexplained voice that inspires him. In this shift, the novel suggests two interrelated problems posed by biloquism: first, that we might not perceive the true source of a voice, since the biloquist can change the direction of his speech; and second that the authenticity of a voice cannot be guaranteed, since the biloquist can produce an accurate imitation of anyone.

Biloquism strikes me as a useful metaphor for considering generative AI because chatbots also have the capacity to “speak” from an unknown source and to imitate the voices of others. Theodore Wieland hears mysterious voices in the air, and while we know the source of a chatbot’s words, large language models pose a similar question of agency. A chatbot is not a human author, but the human using a large language model’s application programming interface doesn’t create the words on the screen, either. And given the vast quantity of data contained in a large language model’s training set, humans don’t have the ability to retrace the decisions within a chatbot’s neural network to understand why the program produced specific content. Such ambiguity underlies speculation that generative AI models might be approaching, or have already achieved, consciousness or sentience, like OpenAI chief researcher Ilya Sutskever’s provocative 2022 tweet that “it may be that today’s large neural networks are slightly conscious.” Sutskever’s qualified claim of slight consciousness gets to the philosophical problem we face in recognizing sentience in artificial intelligence. Lacking a universally accepted definition of human consciousness, how do we devise a universally accepted definition of machine consciousness? 

The primary plot of Wieland dwells in this kind of ambiguity where science and speculation begin to blur. Theodore’s madness and the unexplained voices inspiring him shakes Clara and her family’s belief in a rational universe. Yet the novel’s focus on the line between empirical knowledge and speculative belief draws our attention away from the problem of authenticity and imitation posed by biloquism. That tension within the plot of Wieland offers a heuristic for the more quotidian, rather than existential, problems posed by generative AI.

Like Carwin, chatbots such as ChatGPT can imitate the voices of other writers or speakers. Like Carwin, users of generative AI programs could also falsely create sexually suggestive material or other content designed to embarrass an individual. The spread of pornographic deepfake images of Taylor Swift on social media in January and February 2024 offers just one prominent example of this problem.

Wieland explores Clara’s loss of agency when her friends and family trust the false content Carwin can create more than their prior knowledge of her character. Clara finds herself with few options besides trying to persuade the man who has spoiled her reputation to repair it. It’s at this point in the novel, however, when Clara discovers that her brother has murdered his wife and children, that the plot shifts from concern over Clara’s reputation to concern over her very life. The problem of imitation and the questions of authenticity it poses cede precedence to questions of subjectivity and agency.

Deepfakes and other deceptive uses of generative AI represent an immediate problem, but as with Brown’s novel, we’re often more interested in the mysterious voices. As many AI scholars have noted, contemporary discourses of AI catastrophe, like visions of general AI systems that revolt against humanity, or fears of autonomous weapons systems that evolve beyond human control, tend to minimize human agency and cast AI as a problem for the future. When we focus on looming dangers of AI subjectivity in an imagined future, we also tend to imagine conflicts with a non-human agent. That potential problem can overshadow the existing and more pressing dangers that come from AI misuse by human actors.

Thinking about this problem through the lens of Brown’s novel suggests our tendency to prioritize one side of a two-part problem. Biloquism reminds us that the questions of agency posed by generative AI are also always questions of imitation and authenticity. Biloquism is a human skill, like generative AI is a human product. Brown explains in a footnote defining biloquism that “[t]his power is, perhaps, given by nature, but is doubtless improvable, if not acquirable, by art.” ChatGPT and other chatbots require improvement by the “art” of data labelers (often low-paid workers in the Global South) who flag inappropriate content in the program’s training set.

The novel’s voices and today’s chatbots participate in a discursive context defined by human choices and dependent on human interpretation, regardless of the reality of supernatural voices or the likelihood of machine consciousness. Non-human discursive partners take on only the subject positions that human beings choose to recognize. Brown’s novel asks us to think carefully about that tendency, and how it often enables people to evade accountability for their choices.

Carwin’s reluctance to take full responsibility for his actions at the novel’s climax presents a significant example. Carwin blames his initial choice to test the Wieland family with mysterious voices on the inspiration of his “daemon of mischief.” You don’t have to squint too hard to see the similarities between daemon of mischief and that familiar Silicon Valley buzzword, disruptor, and what better way to describe his effect on the Wielands than Facebook’s old motto of “move fast and break stuff”? Carwin further denies reponsibility when he tries to persuade Clara that he did not give Theodore the command to murder his family:

Catharine was dead by violence. Surely my malignant stars had not made me the cause of her death; yet had I not rashly set in motion a machine, over whose progress I had no control, and which experience had shown me was infinite in power? Every day might add to the catalogue of horrors of which this was the source, and a seasonable disclosure of the truth might prevent numberless ills. . . .

I have uttered the truth. This is the extent of my offences. You tell me a horrid tale of Wieland being led to the destruction of his wife and children, by some mysterious agent. You charge me with the guilt of this agency; but I repeat that the amount of my guilt has been truly stated. The perpetrator of Catharine’s death was unknown to me till now; nay, it is still unknown to me.

In this dialogue, Carwin displaces the burgeoning responsibility he feels for creating a context that would destroy the lives of the Wieland family. Carwin transitions from recognizing that his imitations exacerbated Theodore’s break from reality to placing the blame on an unknown force. The last line of this passage, where he states that he still doesn’t know who killed Catharine, even as Clara has just informed him that it was Theodore, indicates that Carwin seeks to maintain his belief in a power outside human control.

In the novel’s opening paragraphs, Clara tells her unnamed correspondent to: “Make what use of the tale you shall think proper. If it be communicated to the world, it will inculcate the duty of avoiding deceit.” There are plenty of examples of deception within the novel, and Carwin obviously represents an early example of the kind of confidence man that will become so prominent in subsequent American literature. But I don’t think Brown is just urging readers to be more cautious in avoiding scams and grifters here.

Instead, the kind of deceit the novel has in mind is the self-deception that occurs when we cast responsibility outside ourselves and make imagined external subjects responsible for decisions that are ultimately the product of our own cultural choices. Being aware of the possibility of such deceit strikes me as more essential than ever as generative AI becomes part of our classrooms, our workplaces, and our lives more broadly.

 

Further Reading

Emily Bender, Timnit Gebru, Angelina McMillan Major, and Shmargaret Shmitchell, “On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?” FAccT ’21: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (March 2021): 610–623.

Charles Brockden Brown, Wieland; or The Transformation, ed. Philip Barnard and Stephen Shapiro (Cambridge, Mass.: Hackett Publishing Co., 2009).

Kevin Roose, “A Conversation with Bing’s Chatbot Left Me Deeply Unsettled,” New York Times, February 16, 2023.

Oliver Whang, “How to Tell If Your AI is Conscious,” New York Times, Sept. 18, 2023.

 

This article originally appeared in April 2024.

 

---

James M. Greene is Associate Professor of English at Indiana State University, where he also serves as the Faculty Fellow for Artificial Intelligence. He is the author of The Soldier’s Two Bodies: Military Sacrifice and Popular Sovereignty in Revolutionary War Veteran Narratives (Baton Rouge: Louisiana State University Press, 2020).