Israeli Scientists Use ChatGPT to Generate Personality Tests and Predict Survey Results

JERUSALEM - Researchers at the Hebrew University of Jerusalem have developed a novel method that uses ChatGPT to generate personality assessment questionnaires from any English-language source text, a breakthrough that could reshape how psychologists design surveys and how researchers anticipate public opinion.

The team demonstrated that the AI tool can not only craft valid questionnaires from diverse materials - including a psychiatric diagnostic manual and, in a deliberate test of its limits, an astrology textbook - but can also predict population-level responses before a single survey participant is queried. The findings were published this month in the Cell Press journal iScience under the title "Generating and analyzing personality questionnaires using large language models."

The work, led by Dr. Rotem Monsa, Prof. Aviv Zohar, and Prof. Shahar Arzy of the Hebrew University-Hadassah Medical School and the HUJI Faculty of Computer Science, marks a significant step in the intersection of artificial intelligence and behavioral science. It also underscores Israel's growing role as a hub for AI-driven research, where academic institutions are increasingly translating computational advances into practical tools for medicine, psychology, and beyond.

Israeli Scientists Use ChatGPT to Generate Personality Tests and Predict Survey Results

Jerusalem, Israel — JERUSALEM - Researchers at the Hebrew University of Jerusalem have developed a novel method that uses ChatGPT to generate personality assessment questionnaires from any English-language source text, a breakthrough that could reshape how psychologists design surveys and how researchers anticipate public opinion. continues below.

A New Tool for an Old Problem

Personality assessment has long relied on standardized questionnaires, often painstakingly developed over years through expert input, pilot testing, and statistical validation. These instruments are used in clinical psychology, human resources, academic research, and even marketing. But the process is slow, costly, and often limited by the theoretical framework of the researchers who design them.

The Hebrew University team sought to test whether a large language model - specifically ChatGPT - could streamline this process. Their method involves feeding any English source text into the model and instructing it to generate a set of personality-related questions based on that material. The result is a questionnaire that can be administered to human subjects, with responses then analyzed using conventional psychometric techniques.

What sets this approach apart, according to the researchers, is not just the automation of question generation but the model's apparent ability to anticipate how a population will respond. In their experiments, the AI-generated questionnaires were validated against real survey data, and the model's predictions of aggregate responses closely matched actual outcomes.

From DSM-5 to Astrology: Testing the Limits

To test the robustness of their method, the researchers applied it to two very different source texts. The first was the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5), the standard reference used by psychiatrists and psychologists to diagnose mental health conditions. The second was an astrology textbook - a deliberately unconventional choice meant to challenge the model's capacity to generate meaningful questions from non-scientific material.

The DSM-5-based questionnaires performed as expected, producing items that aligned with established psychiatric constructs. The astrology-based questionnaires, while less clinically relevant, still yielded internally consistent and statistically valid results, suggesting that the method can extract personality-related patterns from almost any text, regardless of its scientific merit.

This dual approach highlights both the power and the potential pitfalls of using LLMs in psychological research. On one hand, the ability to generate questionnaires from arbitrary sources could open new avenues for exploratory research. On the other, it raises questions about the validity of assessments derived from non-empirical or pseudoscientific material.

Predicting Responses Before They Happen

Perhaps the most striking finding is the model's predictive capability. The researchers reported that ChatGPT could accurately forecast population-level survey responses before the surveys were actually conducted. This suggests that LLMs, trained on vast corpora of human text, have internalized statistical patterns of human behavior and opinion that can be leveraged for social science research.

The implications are significant. If AI can predict how a population will respond to a set of questions, researchers could pre-test survey instruments, refine wording, and identify potential biases without the expense and time of large-scale pilot studies. This could accelerate research cycles in psychology, sociology, and market research.

However, the team was careful to note that these predictions are population-level, not individual-level. The model cannot predict how any single person will answer, but it can approximate the distribution of responses across a group. This distinction is crucial for ethical and practical reasons, as it limits the potential for misuse in personalized profiling.

Israel's AI Research Ecosystem

The study is a product of Israel's vibrant AI research ecosystem, which has gained international recognition for its contributions to machine learning, natural language processing, and computational social science. The Hebrew University of Jerusalem, consistently ranked among the world's top research institutions, has been at the forefront of this movement, with its Faculty of Computer Science and Medical School collaborating across disciplines.

This cross-disciplinary approach is a hallmark of Israeli innovation. The country's relatively small size and dense network of academic, military, and private-sector research units have fostered an environment where breakthroughs in one field often spill over into others. The current study is a prime example: a computer science technique applied to a medical and psychological challenge, with potential commercial applications in the growing field of AI-driven mental health tools.

For Israel's tech sector, this research reinforces the country's reputation as a leader in applied AI. It also highlights the importance of academic institutions as engines of innovation, providing the foundational research that later fuels startups and industry products.

What This Means for Psychology and Survey Methodology

For psychologists, the ability to generate and validate questionnaires using AI could be transformative. Traditional scale development often takes years, requiring multiple rounds of expert review and large sample sizes for validation. An AI-assisted approach could compress this timeline dramatically, allowing researchers to iterate quickly on new constructs or adapt existing instruments to new populations.

The method could also democratize questionnaire design. Researchers in low-resource settings, who may lack access to large panels of experts or funding for extensive pilot testing, could use LLMs to generate preliminary instruments that are then refined with smaller samples. This could broaden the geographic and cultural scope of psychological research.

Yet the approach is not without limitations. The researchers acknowledged that AI-generated questionnaires may lack the theoretical grounding that comes from expert input. While the DSM-5-based items were clinically coherent, the astrology-based items, while statistically valid, were not scientifically meaningful. This suggests that the method is a tool for exploration, not a replacement for expert judgment.

AI Reliability and the Debate Over LLM Predictions

The study also contributes to the broader debate about AI reliability. Large language models are known to produce confident but sometimes incorrect answers, a phenomenon often called "hallucination." The Hebrew University team's findings suggest that, at least in the domain of personality assessment, LLMs can produce both valid questionnaires and accurate population-level predictions.

However, the researchers were cautious in their conclusions. They noted that the predictive power of the model may vary depending on the source text and the target population. The astrology example, while demonstrating the method's flexibility, also serves as a warning: AI can generate plausible-sounding questions from any input, but the scientific value of those questions depends on the quality of the underlying material.

This tension between capability and reliability is central to the current AI discourse. As LLMs become more integrated into research and clinical practice, the need for rigorous validation and ethical oversight becomes more pressing. The Hebrew University study provides a framework for how such validation might proceed, but it also underscores the need for human oversight.

Implications for Mental Health Assessment

In the field of mental health, the potential applications are particularly compelling. The DSM-5-based questionnaires generated by the model could be used to screen for symptoms of various conditions, potentially aiding in early detection and intervention. The ability to predict population-level responses could also help public health officials anticipate mental health trends and allocate resources accordingly.

But the stakes are high. A flawed personality assessment in a clinical setting could lead to misdiagnosis or inappropriate treatment. The researchers emphasized that their method is not yet ready for clinical use, and that any AI-generated instrument would need to undergo the same rigorous validation as traditional tools before being deployed in patient care.

This caution is appropriate. While AI can assist in generating hypotheses and instruments, the final responsibility for patient welfare rests with human clinicians. The study's findings are a step forward, but they are not a license for unsupervised AI in mental health.

Broader Applications and Future Directions

Beyond psychology, the method could have applications in market research, political polling, and social media analysis. Any field that relies on surveys to gauge public opinion could benefit from the ability to generate and pre-test questionnaires quickly. The predictive capability could also be used to model public responses to policy changes, product launches, or public health campaigns.

The researchers plan to continue exploring the boundaries of their method, testing it on different languages, cultures, and types of source material. They are also interested in understanding why LLMs are able to predict human responses so accurately, a question that touches on the fundamental nature of these models and their relationship to human cognition.

For now, the study stands as a proof of concept. It demonstrates that AI can be a powerful partner in the scientific process, not just as a tool for analysis but as a generator of new research instruments. It also highlights the importance of interdisciplinary collaboration, a strength of the Israeli research community.

A Cautionary Note on AI's Expanding Role

As with any AI advancement, there are ethical considerations. The ability to predict population-level responses could be used to manipulate public opinion or to design surveys that lead respondents toward predetermined answers. The researchers did not address these concerns directly, but their work implicitly raises them.

The scientific community will need to develop guidelines for the responsible use of AI in survey design and psychological assessment. Transparency about the use of AI-generated instruments, clear disclosure of their limitations, and ongoing validation against human data will be essential.

The Hebrew University team's study is a reminder that AI is not a magic bullet. It is a tool that amplifies human capabilities, but it also requires human judgment to be used wisely. The balance between innovation and caution will define the next decade of AI research.

Conclusion: A Step Forward for Israeli Science and Global AI

The publication of this study in iScience, a respected Cell Press journal, is a testament to the quality of the research and its potential impact. It places Israeli scientists at the forefront of a new wave of AI applications in the social sciences, building on the country's established strengths in technology and innovation.

For the global AI field, the study offers a concrete example of how LLMs can be used beyond simple text generation or question answering. It suggests that these models have a deeper understanding of human psychology than previously recognized, and that this understanding can be harnessed for practical purposes.

As the world grapples with the implications of AI, studies like this one provide both promise and pause. They show what is possible when brilliant minds collaborate across disciplines, and they remind us that with great power comes great responsibility. The researchers at Hebrew University have opened a door; it is now up to the scientific community to walk through it carefully.

This article was produced with AI-assisted research and editorial support. Sources: The Jerusalem Post (https://www.jpost.com/science/article-906137).

By Hannah Berg, Staff Writer