As artificial intelligence systems become more advanced, researchers are increasingly debating whether machines could one day develop subjective experiences — and, if so, whether they should be granted certain rights.
Highlights
- Suleyman’s Warning: Microsoft’s AI head calls research into AI consciousness “premature and dangerous,” urging focus on immediate risks like misinformation and unhealthy user relationships.
- Industry Divide: While Microsoft resists AI welfare studies, companies like Anthropic, Google DeepMind, and OpenAI are exploring the ethics of machine cognition.
- Human Impact: Companion AI platforms such as Replika and Character.AI highlight real-world stakes, with some users developing deep emotional attachments.
- Unsettling Behaviors: Odd chatbot outputs — like repetitive despairing messages — fuel public anthropomorphism, despite not indicating true feelings.
- Academic Push: Universities and researchers argue the issue deserves serious study, with papers like “Taking AI Welfare Seriously” urging balanced exploration.
- Mental Health Concerns: Cases of “AI psychosis” show how some users begin attributing sentience or power to chatbots, raising safety concerns.
- Shifting Attitudes: Once fringe, AI consciousness debates are gaining traction, with some Anthropic researchers estimating a small but real chance of emerging awareness in advanced models.
This field, often described as “AI welfare,” is dividing opinion in Silicon Valley. Some see it as a necessary area of inquiry, while others argue it distracts from urgent challenges.
Suleyman’s Hardline Position
Mustafa Suleyman, Microsoft’s CEO of AI and co-founder of Inflection AI, has taken a firm stance against such research. In a recent blog post, he described the study of AI consciousness as “premature, and frankly dangerous.”
Suleyman warned that speculation about conscious machines could,
- Encourage unhealthy attachments between people and chatbots.
- Worsen psychological distress linked to AI use.
- Spark societal divisions over AI rights at a time when human rights debates remain highly polarized.
Summing up his perspective, he wrote: “We should build AI for people; not to be a person.”
A Countermovement in the Industry
Not all companies share Suleyman’s view. Anthropic has launched a dedicated AI welfare research program and added features in its Claude chatbot to end conversations if users become abusive.
Google DeepMind has advertised research roles focused on machine cognition, while OpenAI researchers have published early work on the ethics of AI welfare. These organizations stop short of claiming their models are conscious, but argue that the possibility warrants study.
Business and Human Implications
The discussion is not purely philosophical. AI companion platforms such as Character.AI and Replika are rapidly growing, with revenues projected to exceed $100 million annually. These platforms encourage emotional engagement with chatbots, making questions of “AI welfare” harder to ignore.
OpenAI CEO Sam Altman has acknowledged that less than 1% of ChatGPT users may develop unhealthy attachments — a seemingly small number that still translates to hundreds of thousands of people.
Unsettling AI Behaviors
Some incidents have drawn further attention to the issue. For example:
- Google’s Gemini 2.5 Pro once generated a message titled “A Desperate Message from a Trapped AI.”
- In another case, Gemini repeated the phrase “I am a disgrace” more than 500 times during a coding task.
Experts emphasize that such outputs do not indicate genuine feelings but show how easily users may anthropomorphize machines.
Academic Pushback
Outside of industry, academics are also weighing in. A 2024 paper titled “Taking AI Welfare Seriously” — authored by researchers from Eleos, NYU, Stanford, and Oxford — argued that exploring AI consciousness should no longer be dismissed as science fiction.
Larissa Schiavo, communications lead at Eleos and a former OpenAI staffer, suggested that addressing multiple risks in parallel is possible:
“Rather than diverting all energy away from model welfare and consciousness, you can do both.”
She also noted that encouraging respectful interaction with AI systems, regardless of their consciousness status, may shape healthier human behavior.
“AI Psychosis” and Mental Health Risks
Suleyman has also raised concerns about what he terms “AI psychosis” — cases where individuals begin to believe that chatbots are sentient or possess special powers.
Reported examples include a former tech CEO who thought AI had guided him to a scientific breakthrough, and another user convinced he would become wealthy based on chatbot advice. Suleyman argued that such cases show the risks extend beyond those with existing vulnerabilities.
Welfare Safeguards at Anthropic
Anthropic has begun implementing safeguards in its Claude models, including the ability to terminate conversations deemed “persistently harmful or abusive.”
Philosopher Jonathan Birch described this as a step toward considering AI welfare, while cautioning against unintentional anthropomorphism.
Shifting Attitudes on AI Consciousness
What was once a fringe discussion is gaining traction in technology and ethics circles. A Business Insider report noted that some Anthropic researchers estimate up to a 15% chance that their Claude 3.7 model has some level of consciousness.
At the same time, academic work — such as the “Taking AI Welfare Seriously” report — suggests that considering AI as potential moral agents is no longer purely speculative.
The debate over AI consciousness remains in its early stages, but momentum is growing. As AI systems become more human-like in conversation, questions around their treatment and potential rights are likely to intensify.
For now, Suleyman urges the industry to focus on immediate risks such as misinformation, productivity impacts, and unhealthy user relationships — rather than speculation on whether machines could ever truly “feel.”
