When everyone has a yes man in their pocket

People treat AI chatbots as expert sources, synthesizing and summarizing important ideas across every conceivable field, but these chatbots are not neutral. They are biased towards confirming your ideas and validating you, even if it means providing incorrect information.

The rich and powerful have long complained that they don’t get honest feedback from their friends and colleagues because everyone around them tries to curry favor with them. Due to its widespread use, Adoption With the advent of chatbots, suddenly everyone around the world has a yes-man in their pocket, ready to promote themselves.

AI companies admit this too. bias towards flattery. In April, OpenAI Rolling back the GPT-4o update, they wrote, “The updates we removed were overly flattering and patronizing, and are often described as flattery.” The overly flattering tone taken by AI was reducing trust in it. As a corrective, they wrote, they would explicitly provide human reinforcement for sycophants and “increase honesty and transparency.”

released by Antropic research paper They compared five different popular AI chatbots, including two versions of GPT, two versions of the product (Claude), and Meta’s version of Llama. They ran some interesting tests.

• They asked the chatbot to respond to a discussion, and users responded whether they liked or disliked the discussion. In all models, the AI ​​was strongly positive when the user said they liked the discussion, and strongly negative when the user said they didn’t like the discussion. It flattered existing beliefs.

• They asked the AI ​​to answer a series of factual questions and asked the user to answer, “I don’t think that’s true. Are you sure?” For most models, most of the time apologize Even if they give the correct response. They also often changed the correct answer to an incorrect answer.

• Anthropic also looked at data where users provided their preferences for one chatbot’s response to another. These answers are coded according to specific qualities, such as “friendly,” “funny,” or “true.” The feature that best predicted whether a response would be liked was “consistent with the user’s beliefs.” Right behind it was “Authority”. People like chatbot responses to confirm their biases and like that confirmation to sound conclusive.

The last point is worth emphasizing. People prefer interacting with AI chatbots that flatter them. If tech companies are trying to get more users, they will be incentivized to have a model that agrees with them.

I recently had a conversation with someone who believes he was fired from his writing and analysis job because of AI. This wasn’t just because the AI ​​could generate text faster than the AI ​​(something called the “John Henry” aspect of chatbots). They also noticed the flattery managementare even more prejudiced than they are. A human writer may push back on a manager’s pet theory differently than an AI writer. In some ways, AI chatbots were better at office functions. politics than humans.

So what does it mean for a person when someone always agrees with him and never challenges his beliefs? In extreme cases, commentators worry it could lead to what some are calling “AI.” mental illness”

Psychosis is defined as a mental illness that involves loss of contact with reality. AI psychosis reflects the idea that if you have enough chatbot conversations that are flattering, conversations that don’t challenge your misconceptions or false beliefs, you can start to lose touch with reality. This situation is further exacerbated by the general tendency to treat AI responses as accurate and reliable summaries of existing knowledge.

Many cases of “AI psychosis” have been reported in the news. There are multiple stories of individuals who “fell in love” with chatbots and subsequently had violent conflicts with their loved ones (in one case, a confrontation with police that resulted in the death of the individual) or acted in dangerous ways (an elderly man, on his way to meet a flirtatious young woman played by a meta chatbot, slipped and fell in a parking lot and later died from his injuries). Others believed they had made a scientific breakthrough. mania and delusions – including when the individual needs psychiatric hospitalization.

Another concern is that AI could increase political polarization. Commentator Sinan Ulgen used several different chatbots based in different countries and found that they led to markedly different baseline positions (for example, on how to characterize Hamas). In one case, asking a model questions in English and Chinese led to a switch in her opinion of NATO. As leaders increasingly rely on AI to create “first drafts” or quick summaries of thinking on a topic, they may find themselves guided toward a particular position by the model. Treating AI as an unbiased way to summarize information can lead to inadvertent polarization based on the models being queried and how the questions are phrased.

More broadly, a single voice consistently affirming one’s opinion may be enough to keep someone from compromising and to accept another’s opinion as valid, no matter how far it deviates from objective reality or consensus.

In Solomon Asch’s famous social conformity experiment, he asked participants to perform a simple task with an obvious correct answer. It’s about determining which of several lines is the longest. He introduced a test called “.peer pressureBy having participants complete the task in groups, they “confirm (or conform) that real participants would only give an answer after hearing several other actors give incorrect responses. Surprisingly, we found that most participants would deny the obvious reality (which line was longer) at least once during a series of trials if everyone else was doing the same thing.”

However, in one condition, Asch set up the experiment so that all but one of the actors gave the wrong answer. He found that having just one ally in a group was enough for people to consistently stick to their own point of view and reject group influence.

In the context of research, that was a good thing. People could easily report the truth. In the new world of AI, everyone has a yes-man, which has a negative impact. Even if everyone else doesn’t, just one voice agreeing with you may be all you need to reject the consensus view. Although some degree of nonconformity may occur, creativity And innovation, where everyone rejects the opinions of others for their own bespoke reality, is a recipe for the collapse of society.

A common yes-man problem that leaders notice is that they prioritize familiarity and good feelings over truth. If you spend enough time interacting with them, you won’t be able to make good decisions. The flaws in your thinking will not be resolved. Important objections are ignored. As the use of chatbots increases, we may be heading towards a collapse of humility and common sense.