Surprising: ChatGPT Can Predict Personality and Answers to Human Psychology Tests
OpenAI’s GPT-4 artificial intelligence is not only adept at composing text and answering complex questions. A recent study shows that the AI model is capable of understanding human personality patterns and predicting how someone responds to questions in psychological tests.
The research, published in the journal iScience, was conducted by Rotem Monsa, Aviv Zohar and Shahar Arzy from the Hebrew University of Jerusalem. They tested whether GPT-4, trained on an enormous volume of human text, had learned the fundamental structure of human personality.
In their experiment, the researchers fed material from two different sources into GPT-4. The first source was the DSM-5, the diagnostic manual for mental disorders used by psychiatrists. The second was a popular astrology guidebook titled The Only Astrology Book You’ll Ever Need.
The research team deliberately chose the two books because both describe personality in depth, although only the DSM-5 has years of clinical research behind it.
From those instructions, GPT-4 produced 50 statements based on descriptions of 10 personality disorders in the DSM-5, as well as 60 statements based on the 12 zodiac signs. These questions were then tested on 588 adult respondents in the United States and the United Kingdom.
The results surprised the researchers. The AI-generated questions, even those sourced from the astrology book, were able to predict respondents’ wellbeing, income and education levels on a par with standard psychological questionnaires such as the Big Five Inventory. For education level specifically, the AI-generated astrology-based questions actually performed better than the Big Five questionnaire.
In an interview with Earth.com, Monsa revealed one of the most striking findings. “The astrology result was the biggest surprise,” Monsa said.
Monsa explained that astrology descriptions have essentially absorbed everyday personality language over centuries.
“Astrology descriptions have been culturally refined over centuries and overlap heavily with everyday trait language, which is why the individual questions carry signal. It is the organising framework that does not,” Monsa said.
Beyond compiling questionnaires, GPT-4 was also asked to predict the average response values of the public simply by looking at the sentence formulations. The AI’s estimates were highly precise and deviated only slightly from the respondents’ actual results.
“That is practically useful, but it also shows that these models have absorbed something about how populations respond to personality questions, purely from reading text,” Monsa said.
Nevertheless, the researchers stressed that these results cannot yet be used as an individual medical diagnostic tool. The AI-generated questionnaires only work at the group average level, not for individual personalities.
“I would not judge a questionnaire by who, or what, wrote it. I would judge it by whether the questionnaire has been validated,” Monsa concluded.