Experiment
Published: July 15th, 2026
Last updated: July 15th, 2026
---
It has been argued that giving an LLM an expert persona helps the model to tap into its already large knowledge base from pre-training to produce better answers.1 It has also been noted that it produces more detailed, longer answers.2 But do expert personas actually improve underlying performance when it comes to factual recall and reasoning?
To test this, I conducted an experiment using gemini-2.5-flash-lite (with temperature set to 0) on 500 randomly selected questions from the MMLU-pro dataset. The LLM answered the same questions under a control condition with no persona and an experimental condition with an expert persona.
Despite observing a 1 percentage point increase in accuracy for the expert persona, the inferential statistical methods used here suggest that this was likely due to chance. This is supported by more recent research indicating that expert personas do not reliably improve factual recall and reasoning in LLMs.3 Instead, they appear to be most useful for advisory tasks where answers require structured caution and risk assessment.4
đź’» What I did
The evaluation uses the MMLU-pro dataset loaded from Hugging Face, with questions selected through random sampling. A filtering mechanism excludes questions that have already been answered under the same prompt version and experimental condition, allowing additional samples to be processed across multiple runs using the same random seed.
Each selected question is formatted into a standardised prompt, with the expert condition including an initial instruction identifying the model as an expert in the relevant subject (this line is omitted in the control condition). The model is instructed to return only the letter corresponding to the correct answer enclosed in brackets (e.g., (C)), a format chosen to simplify reliable response parsing and reduce errors caused by additional generated text.
Responses are parsed using strict rules: if only the correct bracketed letter is detected, the response is marked as a pass; if only an incorrect letter is detected, it is marked as a fail; otherwise, it is classified as a parse failure and retried in subsequent runs. After each successfully processed question, the result—including passes, failures, and parse failures—is immediately written as a JSON object to a JSONL file before the next question is processed, ensuring progress is preserved.
đź’ˇ What I found
The expert persona achieved a slightly higher pass rate than the control condition, improving overall accuracy by around 1 percentage point across the 500 randomly sampled questions. While this initially suggests that assigning an expert persona may provide a small benefit, the observed difference varies by category and was seen as likely too small overall to conclude that the effect is genuine.
To confirm this, inferential statistical techniques were applied. Bootstrapping with 10,000 iterations produced a 95% confidence interval for the difference in pass rates that included zero. Bootstraping is when the results are resampled (with replacement) and recomputed to see how much the result can vary. A confidence interval is a range within which we think the "true result" would most likely fall if rerun with new data. The inclusion of 0 indicates that the true effect could plausibly be no improvement at all.
An Exact McNemar's test, which compares paired outcomes from the same set of questions, produced a p-value of 0.55, well above the conventional threshold of 0.05 for statistical significance. The exact McNemar's test evaluates whether the observed disagreement counts (where a question was passed under one conditon but not another) are likely to have occurred by chance by chance alone. If the expert persona has no real effect, improvements and declines would be expected to occur at similar rates.
These results suggest that, under the conditions tested, the apparent improvement is most likely explained by chance rather than a genuine effect. However, the findings should be interpreted in the context of the experiment's limitations. They include the use of a well-known benchmark that my have caused data contamination (when a model remembers answers from it's training data), only 500 unique questions, a single run per question, a relatively simple expert persona and a single model.
Dig deeper
- GitHub repo:
llm-expert-persona-eval - Jupyter Notebook:
Expert persona eval analysis
Sources
- Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm. arXiv
- ExpertPrompting: Instructing Large Language Models to be Distinguished Experts. arXiv
- Expert Personas Improve LLM Alignment but Damage Accuracy: Bootstrapping Intent-Based Persona Routing with PRISM. arXiv
- When Does Persona Prompting Actually Help? A Retrieval and Metric Analysis of Expert Role Injection in LLMs. arXiv