While you may expect your favorite chatbot to offer the same responses irrespective of the language you use, a new study published in the Royal Society Open Science journal (via Tech Xplore) suggests otherwise. Researchers from the University of Science and Technology of China have found that popular AI chatbots exhibit different personality traits depending on the language of the prompt.
Standardized personality tests reveal shifting trait profiles
The researchers put five chatbots through the Big Five Inventory-2 personality questionnaire in different languages. The lineup included ChatGPT-4o, Gemini 2.0 Flash, DeepSeek-V3, Ernie 3.5, and Le Chat. Each model answered the 60-statement questionnaire in English and Mandarin, while Le Chat was also tested in French.
Across 30 runs for each language, the researchers found that the chatbots generally gave stable and favorable self-descriptions, often stressing cooperativeness and dependability while reporting very little negative emotionality. However, there was a measurable shift in the scores based on the language picked. ChatGPT-4o scored 4.74 out of 5 on open-mindedness in English and 4.81 in Mandarin. DeepSeek-V3 went the other way, landing at 4.83 in English and 4.81 in Mandarin.
Gemini 2.0 Flash also stood out, posting higher conscientiousness and lower negative emotionality in Mandarin. The team also compared the chatbot profiles with human survey data fom the US, China, and France, and no model fully matched its local reference group.
Anthropic saw something similar in its own research
The results line up with what Anthropic has already said about its own chatbot. Earlier this year, the company confirmed that Claude acts differently depending on your language and which model you pick. In its study, Anthropic looked at how Claude behaves in real conversations, while this study used a standardized questionnaire across several rival models. Since both reached a similar conclusion, the pattern is harder to dismiss as a one-off.
Both sets of researchers are careful about what this means, though. The new study shows how models respond to prompts and not that chatbots have real personalities. It also couldn’t separate the effect of language from training data, model design or how the translated statements were interpreted. Anthropic has admitted it doesn’t yet know whether these shifts are a flaw or simply the model adapting to different cultural norms. For bilingual users, the practical takeaway is that asking for a second opinion in another language may get you a different answer and not just a translation.
RELATED COVERAGE: