Dynamic Multi-Agent Sampling with LLMs: Simulating Large-N Survey Participants with Open-Source Tools


Read time

7 min read

Published date

 Feb 9, 2026

Category

Machine Learning

Weekly Newsletter Update!

Stay up-to-date with the latest innovations, features, and tips in no-code website building!


Dynamic Multi-Agent Sampling with LLMs: Simulating Large-N Survey Participants with Open-Source Tools

This project introduces a multi-agent system for replicating real-world survey outcomes and identifying systematic limitations of RLHF-aligned language models in opinion simulation. We deploy locally hosted, open-source large language models—specifically Allen AI’s OLMo-2-32B in both instruct-tuned and base configurations—to dynamically generate simulated survey participants. The system introduces Extraction Augmented Orchestration (EAO), adapting Google’s LangExtract for structured context injection and programmatic survey administration. We source demographic and psychographic data from the 2021 Canadian Election Study (CES) to generate 15,069 simulated respondents, each grounded in the profile of a real CES participant and administered a 45-question post-election instrument. Simulated responses are compared to actual CES returns as validation. A secondary finding concerns the interaction between safety fine-tuning and opinion simulation: instruct-tuned models exhibit complete distributional collapse on politically sensitive items, producing uniform neutral responses regardless of demographic conditioning, while base models avoid this collapse but require carefully structured prompts. This work demarcates the current boundaries of LLM-based survey simulation for generating synthetic public opinion data, pre-testing instruments, and advancing methodological scholarship in computational social science. 

Create a free website with Framer, the website builder loved by startups, designers and agencies.