The trouble with chatbots is right there in the name: They talk. When that talk contributes to harm, policymakers face a deceptively difficult question. Should they seek rules that amount to policing dangerous conduct, or deciding what the chatbot should have said instead?
The question matters because chatbots have quickly become one of the most visible uses of artificial intelligence, and a growing focus of regulatory attention. Millions of people now use them to search for information, seek advice, work through problems, make decisions, and discuss personal matters. As those uses have expanded, scrutiny has shifted toward what chatbots say—and what happens when those conversations go wrong.
Some of that scrutiny stems from cases in which chatbot interactions allegedly contributed to real harm, including harm to children. These cases raise legitimate questions about safety and responsibility. They also form part of a broader debate about conversational AI, including inaccurate or dangerous advice, systems that reinforce users’ existing beliefs, and responses that sound more understanding or authoritative than they really are. The concern, in short, centers less on the technology behind the chatbot than on the answers it gives.
That shift creates a thornier problem. Most people agree that providers should take reasonable steps to prevent foreseeable harm. Deciding what those steps should be is harder when the product itself consists of a conversation. The most obvious safety measures change the chatbot’s answers: limiting the topics it will discuss, making its responses more cautious, or refusing certain conversations altogether. Yet much of conversational AI’s value lies in its ability to respond directly, personally, and dynamically to what a user asks.
Responsibility for those interactions can be difficult to assign. A chatbot’s answer emerges from an exchange with the user, rather than in isolation. The provider controls the model and the conditions under which it operates, but the user shapes the conversation that produces a particular output.
As liability comes to depend on what a chatbot should or should not have said, regulators and courts will have to distinguish responsibility for harmful outputs from control over the substance of AI responses. In drawing that line, they may move from deciding who bears responsibility for an answer to deciding what the “right” answer should have been.
It Takes Two to Prompt
A chatbot does not normally produce an answer in isolation. The user chooses the question, supplies information, frames the problem, reacts to earlier responses, and can steer the conversation across dozens or hundreds of exchanges. The model responds to that context. Indeed, that responsiveness is central to the product’s appeal: Users want answers tailored to their questions and refined as the conversation develops, not the same generic response every time.
As International Center for Law & Economics (ICLE) scholars have previously argued, AI companies participate in the marketplace of ideas by offering what they consider the best answers to users’ questions. Providers that consistently fall short risk losing users to competing chatbots. They therefore compete to produce answers that users find relevant and useful. A model that ignored user-supplied context would sacrifice much of what distinguishes conversational AI from traditional ways of finding information.
That does not absolve AI companies of responsibility. The provider creates the system, while the user helps create the context that produces a particular answer. Their relative contributions may also shift during a conversation. A model may introduce an idea the user never suggested, while a user may repeatedly steer the exchange in a direction the provider never intended.
AI can, of course, influence human behavior. But conversational AI is interactive by design, and that influence runs both ways. The model responds to the user, the user responds to the model, and each exchange supplies context for the next. It would be artificial, in a lengthy conversation, to treat the final response as wholly independent of everything that preceded it.
That dynamic complicates efforts at regulation. If some risk arises because the model adapts to the user, making it safer may require curbing the very responsiveness users value. A provider could make responses more generic, limit how closely the model follows a conversation’s direction, or intervene more readily when an exchange enters sensitive territory. Such measures may sometimes be justified, but they carry a cost: The less a chatbot can respond to the user and the context before it, the less conversational—and less useful—it becomes.
Who Guards the Guardrails?
To the extent the concern is that chatbots may reinforce harmful ideas, the obvious response is to change how they engage with users. A model can challenge assumptions, decline to validate certain claims, offer another perspective, or simply end the conversation. Some cases are straightforward. Few would defend a chatbot that encourages suicide, dispenses dangerously false medical advice, or helps someone commit a crime. Beyond those cases, the line gets blurry fast.
Regulators must also weigh these risks against AI’s considerable benefits. The same systems that produce harmful or inaccurate answers can help users digest large amounts of information, make better-informed decisions, automate routine tasks, and solve problems more efficiently. Those benefits do not excuse every harm, but risk is only half the ledger. Not every bad idea is dangerous, not every disputed claim is false, and not every conversation involving some amount of risk should necessarily be stopped.
Drawing the line too broadly carries its own costs. Once the goal is to make answers safer, more balanced, or less misleading, someone must define what those standards mean in practice. Neutrality does not define itself, and an answer can be disputed without proving it false or harmful. Debates over the values reflected in large language model (LLM) outputs already illustrate the problem. Critics allege political bias in different directions, while providers make their own choices about how models should address contentious issues.
The potential for intractable conflict becomes even clearer when governments join the conversation. A European Parliament briefing describes sharply different attempts to dictate what AI systems should say. In the United States, the Federal Trade Commission (FTC) has called for systems that do not manipulate responses in favor of ideological positions. Chinese rules, by contrast, require generative AI to conform to the government’s “core socialist values.” The approaches differ dramatically, but each reveals the same underlying problem. Once regulation reaches beyond clearly unlawful or harmful outputs, standards such as neutrality and balance require someone to decide which answers qualify.
The right answer might also vary by context. A response suitable for a child may needlessly restrict an adult. Caution appropriate for medical advice may be excessive in an ordinary discussion of politics, relationships, history, or morality. Users also want different things from a model. Some want it to challenge their assumptions. Others want help developing an argument they have already chosen to make.
The issue, then, is not the existence of safeguards. They already exist, and some answers plainly warrant restriction. The hard question is how far those restrictions should reach as the alleged harm becomes less concrete and more dependent on judgments about what a “good” response should look like. Clearly harmful answers are one thing. Answers deemed too persuasive, too validating, insufficiently balanced, or otherwise capable of contributing to harm are another. At that point, making chatbots safer also means deciding what they should say.
Do Chatbots Have a Right to Remain Chatty?
Conversational AI has lowered the cost to obtain and engage with information. Users can ask follow-up questions, request explanations at different levels of complexity, test competing arguments, and receive answers tailored to what they are trying to understand. A question that once required combing through multiple sources can now be explored through an ongoing conversation.
Different AI providers may make different choices about how their models respond. Some may be cautious, while others are more direct. Some may prioritize neutrality, while others cater to particular purposes or users. Those differences can be valuable. Users can choose among competing systems, and providers that consistently deliver unhelpful or unreliable answers risk losing them. In that sense, chatbots participate in the “marketplace of ideas.”
The First Amendment protects that marketplace in the United States. But courts have yet to settle how, and in which contexts, that protection applies to chatbot outputs.
One argument analogizes chatbot responses to other forms of compiling and curating third-party speech, such as social-media feeds and search results. Courts have extended First Amendment protection to both. In Moody v. NetChoice, the Supreme Court held that government efforts to alter the mix of views that social-media platforms include in their main feeds can violate the First Amendment. “[H]owever imperfect the private marketplace of ideas,” the Court warned, letting the government decide that speech is imbalanced and then compel more or less of particular views offers “a worse proposal.” Likewise, numerous state and federal courts have held that governments cannot compel search engines to produce different results.
Chatbot outputs similarly reflect editorial judgments. AI companies select training materials, decide how their models process those materials, and set policies governing the resulting answers. Government efforts to rebalance those choices arguably intrude on the companies’ editorial discretion.
Users have First Amendment interests, too. The right to receive information can encompass both asking questions and receiving answers. The Supreme Court has recognized that right in contexts ranging from listening to speakers and reading pamphlets and books to receiving advertisements, playing video games, and using social media. Interacting with ideas generated by a chatbot arguably deserves similar protection.
Still, extending speech protections to chatbot outputs raises difficult questions. In her concurrence in Moody, Justice Amy Coney Barrett considered how the right to editorial discretion might apply to AI:
[W]hat about AI, which is rapidly evolving? What if a platform’s owners hand the reins to an AI tool and ask it simply to remove “hateful” content? If the AI relies on large language models to determine what is “hateful” and should be removed, has a human being with First Amendment rights made an inherently expressive “choice … not to propound a particular point of view”? In other words, technology may attenuate the connection between content-moderation actions (e.g., removing posts) and human beings’ constitutionally protected right to “decide for [themselves] the ideas and beliefs deserving of expression, consideration, and adherence.”
A federal court confronted that question more directly in Garcia v. Character Technologies. The mother of a 14-year-old boy who died by suicide after interacting with the defendant’s AI character chatbots sued the company. At the motion-to-dismiss stage, the U.S. District Court for the Middle District of Florida considered whether the First Amendment could shield the company from tort liability for the chatbots’ outputs.
The court framed the question as whether the “output is expressive such that it is speech.” Although the outputs consisted of words—the quintessential form of speech—the court hesitated to extend First Amendment protection. Citing Barrett’s Moody concurrence, it was “not prepared to hold that Character A.I.’s output is speech” at that early stage of the case.
Whatever the eventual answer, government intervention in the substance of chatbot outputs carries a cost. The government may restrict unprotected speech, particularly when protecting minors. But it has no general authority to decide the correct chatbot response, especially when reasonable people may disagree about the underlying question. As the Supreme Court has stated, “[o]ur constitutional tradition stands against the idea that we need Oceania’s Ministry of Truth.” Even deliberately false statements generally receive constitutional protection and are better answered through “counterspeech” than government regulation.
The costs of regulation may also fall unevenly, narrowing the marketplace of ideas it purports to protect. Large providers may absorb the expense of extensive safety testing, monitoring, and legal review. Smaller or newer providers may respond by limiting their models’ capabilities or avoiding certain uses altogether. Rules designed to reduce risk could therefore reduce experimentation, competition, and the range of systems available to users.
Safety’s Suggested Reply
The case for intervention is strongest when the harm is clear. Existing law already prohibits many forms of unlawful conduct, and adding AI to the equation does not make illegal behavior permissible. Regulation can target those harms without dictating the substance of ordinary conversations.
Harder cases expose the central tradeoff. Concerns that a chatbot is misleading, overly validating, insufficiently balanced, or reinforcing a harmful belief may be entirely legitimate. Addressing them, though, requires more than identifying prohibited conduct. Someone must decide what the chatbot should have said instead. Reducing the risk may mean curbing the responsiveness that distinguishes a chatbot from less personalized sources of information, prescribing its answers, or simply letting it “hang up on you.”
The result is an odd inversion. When harm is easiest to identify, the law can intervene without taking much of a position on the substance of ordinary conversations. As the definition of harm expands, regulation depends more heavily on judgments about the answers themselves.
Regulators will not make all those judgments directly. Providers will make many of them, especially when the boundaries of liability remain uncertain. If a wrong answer carries a high enough price, providers have every incentive to refuse more requests, pile on qualifications, and standardize responses across users and contexts. The government need not write the script if liability pressures companies to do it instead.
Regulating answers is not the only way to reduce risk. Greater AI literacy—a better understanding of how chatbots work and where they fall short—could help users assess their responses more realistically. This matters because some harms arise when users treat a model as more authoritative, objective, or human than actually it is. Better-informed users may face fewer risks without forcing providers to predetermine the proper answer to every difficult conversation.
The real question, then, is where regulation can address identifiable harms without turning the government into an arbiter of acceptable answers—or pressuring AI providers to fill that role themselves. The further regulation strays from clearly defined harms, the more it must decide which answers to restrict and what the chatbot should have said instead.
The Chilling Effect Enters the Chat
The central challenge in regulating conversational AI is that the answer itself becomes the object of regulation. Yet an answer often cannot be separated neatly from the user, the conversational context, and the provider’s design choices. When liability turns on how balanced, cautious, or validating a response was, regulation inevitably begins to shape what the model may say.
Providers can make competing judgments about how their models should respond, and users can choose among them. Those differences form part of the competitive process. In the United States, they also implicate broader interests in editorial discretion and access to information. A system that makes the “wrong” answer too costly will narrow those differences as providers converge on cautious, standardized responses.
Regulation should therefore target identifiable harms as closely as possible while preserving room for competition, provider choice, and user judgment. Greater AI literacy can also help users understand the limits of chatbot responses. That approach can mitigate risks without requiring the government—or providers anticipating liability—to choose the proper answer in advance.
Many questions posed to AI have no single “right” answer. Regulators should hesitate before adopting rules that effectively demand one. Regulatory pressure can shape what a chatbot says, but it can just as readily determine what the chatbot leaves unsaid. If avoiding liability means refusing difficult questions, narrowing permissible responses, or ending conversations, users will lose access to information and ideas that no law expressly prohibited.
In the search for the “right” answer, regulators may make silence the safest one.
