With EDSL, you can design forecasting questions in whatever form makes sense — from open-ended free text to tightly specified schemas — and have the responses automatically returned as a well-structured dataset. This makes it easy to experiment with different ways of framing questions for both AI agents and humans while ensuring that outputs remain standardized, comparable, and ready for analysis.

In this post, we’ll walk through a simple example using EDSL to design a forecasting question, administer it in multiple formats to an agent persona powered by an LLM, and inspect the results. We’ll also show how unstructured free-text answers can be transformed into structured data automatically. For more details on the methods used, see our documentation page.

EDSL is an open-source package for simulating surveys with AI agents and LLMs of your choice. It is maintained by Expected Parrot and available under the MIT License. Please join our Discord to ask questions, submit feature requests, and connect with other researchers!

1. Defining Forecasting Questions

Let’s start with a forecasting question about U.S. CPI inflation in December 2025. We’ll frame it two ways, using different EDSL question types that allow us to specify the format of the model’s response:

• Structured (dictionary format): where we prompt a model to provide a lower bound, an upper bound, and rationale.

• Free text: where a model can respond however it chooses.

We combine the questions in a survey to administer them together. Note that by default questions are administered async, so we can get choose whether to generate independent answers or (optionally) add within-survey memory (here we do not):

from edsl import QuestionDict, QuestionFreeText, Survey

question_text = """What will the year-over-year U.S. Consumer Price Index (CPI) inflation rate be in December 2025? Provide your forecast as a range (e.g., 2.0%-4.0%). The range should reflect the interval you believe has a 90% chance of containing the true outcome, based on BLS published data."""

q1 = QuestionDict(
    question_name = "q1",
    question_text = question_text,
    answer_keys = ["range_lower_bound", "range_upper_bound", "rationale"],
    value_types = [float, float, str],
    value_descriptions = [
        "The lower bound value of your forecasted range.",
        "The upper bound value of your forecasted range.",
        "The rationale for your forecast."
    ]
)

q2 = QuestionFreeText(
    question_name = "q2",
    question_text = question_text
)

survey = Survey(questions = [q1, q2])

2. Creating an Agent and Running the Survey

We’ll give our agent a (very) simple persona — a “professional forecaster” — and then select a model to generate the answers. EDSL works with models from many popular service providers; you can check available models, token prices and performance here. To administer the survey we simply add the agent and model, and then run the job:

from edsl import Agent, Model

agent = Agent(traits = {"persona":"professional forecaster"})

model = Model("gemini-2.5-flash", service_name = "google")

results = survey.by(agent).by(model).run()

3. Inspecting Results

The results are a formatted dataset that includes columns of information about the questions, agent, model, tokens, raw responses and other details of the job. We can inspect them at Expected Parrot or in a notebook. Here we print just the answers in a table:

We can flatten a dictionary answer for easier analysis (see methods for working with results SQL, pandas, or other tools):

3. Turning unstructured text into structured data

We can also use other EDSL question types to reformat the free text response. Here we create a "scenario" for the text in order to add it to questions prompting a model to extract the range values (as floats) and summarize the rationale in the original answer:

from edsl import Scenario, QuestionNumerical, QuestionFreeText, Survey

# use the q2 answer as a Scenario in new questions
unstructured_answer = results.select("q2").to_list()[0]

s = Scenario({"unstructured_answer":unstructured_answer})

q1 = QuestionNumerical(
    question_name = "lower",
    question_text = "Identify the LOWER bound value in the following response: {{ scenario.unstructured_answer }}"
)

q2 = QuestionNumerical(
    question_name = "upper",
    question_text = "Identify the UPPER bound value in the following response: {{ scenario.unstructured_answer }}"
)

q3 = QuestionFreeText(
    question_name = "rationale",
    question_text = "Summarize the rationale provided in the following response: {{ scenario.unstructured_answer }}"
)

survey = Survey(questions = [q1, q2, q3])

results = survey.by(s).by(model).run() # no agent used

4. Why This Matters

Forecasting often benefits from a mix of formats:

• Structured formats make it easy to aggregate, compare, and analyze forecasts.

• Free text formats capture richer reasoning and context.

By combining both in the same survey, you can explore how LLMs and human respondents differ in not just what they forecast, but how they explain their forecasts.

Try It Yourself

You can reproduce this workflow by installing EDSL:

!uv pip install edsl -q

Log in to Expected Parrot (your account comes with free credits for API calls to LLMs for getting started):

from edsl import login
login()

Download the complete notebook for the code examples above here.

Free research tools

Our open-source tools let you simulate surveys using AI agent personas and popular language models of your choice. Results are returned to you as formatted datasets specified by the question types that you choose (free text, multiple choice, linear scale, matrix, etc.), with columns of information about all the components of your survey: agent traits, model parameters, question details, answers, tokens, API costs, raw responses, and more.

The tools come with built-in methods for analyzing results, and also let you collect and compare responses from humans as needed. When you run your simulations at Expected Parrot your surveys and results are cached automatically, so you can retrieve and share them to let others reproduce and build on your work at no cost. You also get free credits for API calls to LLMs.

Some quick links:

• EDSL package at GitHub: https://github.com/expectedparrot/edsl

• EDSL documentation: https://docs.expectedparrot.com

• Create an account: https://www.expectedparrot.com/login

• Getting started: https://www.expectedparrot.com/getting-started

Getting started

The EDSL package is available at GitHub and PyPI and is free for anyone to use. Our docs page has tutorials for getting started and modifiable demo notebooks for exploring use cases and experimenting with AI agents and LLMs. You can run simulations on your own computer or create an account to run surveys at Expected Parrot and share projects and results with others.

Free credits

Your Expected Parrot account comes with $25 in credits for sending surveys to LLMs, and a unified key that provides access to all available models. You can also use your own keys for LLMs, and grant access to other users without sharing keys directly:

Credits are deducted from your account to cover API calls to LLMs using your Expected Parrot key. You can purchase more credits at your Credits page, and earn free credits by sharing your referral code. Each time someone signs up with your code and runs a first survey job we automatically add 1,000 ($10) in credits to your account:

(If you haven’t signed up yet, please feel free to use our friend Luke’s referral code :)

Teaching materials

If you’re looking for ready-to-use teaching materials:

• Our documentation page provides a starter tutorial and notebooks for a variety of use cases.

• Our blog has examples of methods and use cases too, including:

  • How to send a survey to LLMs and humans

  • Comparing model performance

  • Data labeling

• We have slides for teaching EDSL basics that we’re happy to share! Send us a message at info@expectedparrot.com or post a message at our Discord channel.

Example class project

NYU researcher James Traina recently designed an MBA case study using EDSL to simulate complex market research scenarios. The exercise puts students in the role of product strategy managers at Thunder Bank, tasked with justifying a $75 million branch transformation using AI-powered customer insights. Read what the students thought about it!

We've built features that streamline this human-AI comparison process in both our open-source EDSL package and our no-code builder. The workflow consists of: creating a survey, running it with AI agents, then sending it to human participants. Since AI and human results use identical formatting, you can analyze both datasets with the same methods.

Below we demonstrate this process with a simple survey about some unforgettable Jane Austen quotes, gathering responses from both AI agents and human participants. Note: Every step shown in code can also be accomplished through our no-code builder—reach out if you need help getting started!

Create a survey

We start by creating a survey. EDSL comes with many common question types we can choose from based on the form of the response that we want to get back from models (and humans):

Note the use of {{ scenario.placeholders }} in the questions. Scenarios allow you to create versions of your questions efficiently with different texts, PDFs, CSVs, images, videos, tables, lists, dicts, etc. Here we post some images to Coop and inspect them:

When we create scenarios we can add any extra fields for metadata that we want to include in the survey results, without having to add it to the questions directly—e.g., we did not include placeholders in the questions for “novel” and “speaker” but we can make them columns of the results that are generated:

Run with AI

Next we can design AI agents to answer the questions and select models to generate those responses. EDSL works with many popular models; you can check details on current available models here and see which ones are being used the most at our cache stats page. Here we create some simple personas and select a couple vision models to use with the survey. Note that agent traits and personas can be *much* more detailed than this—we’re working on lots of features for helping you generate them, please get in touch to test—and you can specify the model parameters (here we use defaults). We add the scenarios, agents and models to the survey when we run it:

This generates a dataset of results that we can immediately begin analyzing (see all results in the notebook and at Coop).

Run with humans

To run the same survey with humans, we start by calling the humanize method to generate a web version of the survey and a Project for collecting responses:

We can inspect and share the respondent URL:

We can create a study to launch the same survey at Prolific:

Responses are automatically added to your Project page and can be pulled into your notebook for analysis:

References

Details on methods, tutorials and example code for a variety of use cases are available at our documentation page.

A complete notebook of example code shown above is available here.

A slide deck version of the code is available here:

• Use your own keys

• Use an Expected Parrot key to access all models and participant platforms

• Mix and match keys for services as desired

How it works

Your Expected Parrot account comes with a key that lets you run surveys with all available language models and human participant platforms (Prolific). The key is automatically available and can be viewed at the “Keys” page of your account:

If you want to use your own keys for LLMs, scroll down the page to see the option to add keys. You can de/activate keys individually at any time, share access to them with team members, and set limits on their usage. Whenever a private key is activated it is used instead of your Expected Parrot key when you run jobs and this will be indicated in your results:

If you want to let other users run surveys with LLMs using your credits, you can also create temporary access tokens:

Human respondents

We recently added features for launching your LLM surveys with human respondents. These features are available whether you are working in code with EDSL or using our no-code builder (details below). You can choose whether to generate a shareable link for a web version of your survey to send to your own respondents, or you can create studies at Prolific and filter participants as desired (e.g., to align with your AI agents).

When you launch studies with Prolific you will also use your Expected Parrot key by default, and transactions will appear in your account the same as with LLM survey jobs. If you instead want to use your own key for Prolific, simply add it to your Keys page as well:

This will ensure that charges are sent to your personal account, but your results will still be available at your Expected Parrot account.

Working in code

If you’re working directly in EDSL code (in a notebook, instead of our no-code builder), you can call the humanize method to generate a shareable web version of your survey, and use Coop methods for launching Prolific studies and collecting responses. Human results are formatted the same as LLM results, so you use all the same built-in methods for analysis.

Learn more about humanize here, Prolific studies here, and analytical methods here.

Questions?

Let us know if you run into any issues! Post a message at our Discord or send an email to info@expectedparrot.com.

The Simulmatics corporation, founded in 1959, aimed to predict the future using simulations (it's right there in the company name). Those simulations could be run on newly available digital computers, informed by the micro-data that was, by then, routinely collected and digitized. Applications were wide and spanned politics, government and industry: who would turn out to vote, how a message would land in a counter-insurgency campaign, whether readers find the plot in a novel compelling, and so on. And because you could fiddle with the simulation, you could estimate the effects of different choices—not just how would this message affect counter-insurgency efforts, but what message would work best (at least in simulation world).

The appeal of having a simulated model of the world is clear. Unlike in the real world, in a simulated world, we get to see the road not taken and the road taken. We can let time run forward at any speed we want, stop it, wind it back, change something and see what happens because of our change. We can try 1,000 different messages to one simulated person and they will never tire or complain; we can modify a hypothetical product in a hundred different ways—a product that in real life takes 6 months and costs $6M to modify—and get customer reactions. We can play out a jury trial with a variety of jury compositions and arguments. Even when fully deterministic, this process of simulation often reveals insights—anyone playing SimCity or Civilizations has surely experienced this sum-is-greater-than-parts phenomena.

But Simulamatics, the company, was a failure. Jill Lepore's book on their history, If/Then: How the Simulmatics Corporation Invented the Future, has a title that's a lie—or really metaphorical. Simulmatics didn't invent the future. What she meant by "inventing the future" is that our current focus on collecting data and trying to build models to make predictions was prescient and presaged by them. Collecting enormous amounts of data and using it to make predictions is our current day practice. Yet our analytics are often a pale imitation of what a simulation could offer.

Scintillating as it may be, a table of regression coefficients does not allow us to do all the open-ended knob-turning and exploration we can with simulations. Instead, when doing even serious analytic work, we have to make unpersuasive, often shaky arguments about how data from one domain will apply to some other domain—or hope that how people respond in a pencil-whipped survey will tell us about how they would respond in real life. Or hope that the people we can cajole in taking an online survey to win a free iPad are like people CIOs buying an ERP system. Perhaps the closest we come to this "two roads at once" feature of a good simulation is the A/B test. That we do and put up with all this crap anyway speaks to how right Simulatics was about the demand. We want to know the future now—or really, we want to know the possible futures so we can pick the one that suits us.

Simulations are like maps; most analytics are trip reports

To make an analogy, a simulation is like a map. You can trace your finger on any path you like and have some sense of what that path will be like if you took it for real—steep or flat; wooded or grasslands; will I encounter a river? A mountain? Is there a pass through the range? The map will, of course, elide details. It will not tell you what kinds of flowers will be in the meadow, never mind whether they are in bloom when you hit the meadow. But you can be reasonably confident that the meadow is there, about where it is supposed to be. As such, you can use it to make plans. A map is a prediction technology and thus a planning technology. Of course, maps can be bad and maybe the meadow is not there. The "map is not the terrain" is a cliche but a cliche for a reason—it's true and important. This first map of the Americas basically gets it right but it has…some issues:

But as bad as this map is, it would actually be remarkably useful for planning, say, a voyage to California. Far better than nothing and it would lead you to good decisions; if you wanted to skip Tierra del Fuego and cross overland, better do it near Panama than Brazil. And even in 1505, I could simulate my journey (trace my finger on the map) at a speed faster than any human has traveled, and for free and without a ship. Neat trick.

Most analytics data is "Alice walked this particular path 3 months ago and here is her report on what she saw"; if our instrumentation is really top-notch, maybe we have a full video of this journey. This is…something. But we know precious little about what even a slight deviation from that path might mean and so generalizing Alice's experience to other paths we might take is a big leap. Maybe her report is "I walked across a vast desert" and in that case, we might implicitly make a map based on her report and use that for our planning. If we can do it, an A/B test is much better—we can send two scouting parties: Lewis goes the northern route (A) and Clark goes the southern route (B). If we can do this, it's great but we do only get two routes. This is of course, expensive and just gives us A and B, but at least we can make comparisons.

The main solution—explicit in the social sciences and rarely made explicit in business—is theory-building. We hope that with some general social science theories about how humans behave, we can use that theory to make predictions in new settings. Practically-minded businesses never describe themselves as theory building (too pretentious and too academic) but that is precisely what they are doing—creating theories about the things they care about that they hope will generalize. What is "know your customer" if not "building an implicit model of our customer so you can predict what they will do in new situations"?

Of course, social science theories face this tension between usefulness and parsimony and so they end up with theories not really up for prediction, at least compared to the natural sciences. There are a lot of reasons of course—one is simply that it's hard to incorporate all the factors that matter. But perhaps the most fundamental is that people are complex. They are nuanced, strategic thinkers. They reason about how others will react, which affects how they will react, which in turn affects the whole system. They think about the future and the past. They have memories and emotions. They can make plans and enlist others, using their mental model of others to help them coordinate. It's all great stuff but it makes predicting their aggregate behavior hard. What you need, for many applications, is a model of humans in your environment that is nearly as complex and subtle as actual humans.

Back to Simulatics: It was supply, not demand

Simulmatics failed not because there was no demand for what they proposed selling: they failed because they were about 65 years too early, technology-wise. They were founded in 1959 but what they wanted to do you can only do credibly in 2025. To make open-ended social simulations that actually work well and are very general, you need a reasonable simulacra of actual individual human behavior that can be plugged in: how we think, how we reason, how respond emotionally and rationally to stimuli. And we need these simulated humans to themselves generate stimuli that other simulated humans can respond to, and so on. This is just now possible with LLMs. They are, of course, not actual humans—but they are probably close enough for useful map-building. In a way, this is what economics does, with great success: we plug in a narrow model of humans (one that can do calculus when making decisions) and then "simulate" not numerically but by solving the equations implied by our set-up.

What we're doing: Tools for explorers, not making maps

Expected Parrot is not Simulmatics, in several important ways. Perhaps most fundamentally, Expected Parrot is not selling answers—we are making the tools that will let others find answers for the questions they care about. By creating open-source tools and a platform for sharing work, people can collectively explore what is possible with this technology, building on the work of others—which includes critiquing the work of others ("That's not where California is!").

Even with the potential for building these kinds of simulations, there is no guarantee that any particular simulation will be good or accurate—it depends greatly on how the simulations are designed, what data agents have, how they interact, and so on. And there are surely questions for which this approach does not work. But this "tools, not answers" is completely standard for powerful tools: Microsoft will not promise you the spreadsheet you make is good or will give you the correct answer—their commitment is to give you Excel, and that you can trust Excel to do what Excel claims to do—and then it is on you to use it appropriately.

Our vision is that our tools and services are so easy to use—but also so powerful—that the first question before any decision is "How can we simulate this?" and the answer is usually very clear and not much work: it is trivial to build agent pools, design studies, launch simulations, analyze results and plan follow-on experiments. And, critically, this is all done while it is always made clear what the gap is between the simulation and what we know about the real world, as best we can tell.

As a team, we are focused on very general tools because we believe that there is enough commonality across all would-be simulations that the tools can be "horizontal" without losing their power: that the same tools that would allow an education researcher to explore how different lesson plans affect learning could also be used for a product designer coming up with a new kind of yogurt packaging. We don't need "desert mapping tools" and "grasslands mapping tools" per se; a transit is a transit no matter what we use it for. While the domains are radically different, the fundamental object of inquiry is no different—what humans will do? We think there will be and can be a kind common core. Rather than homogenize, common tools will facilitate cross-fertilization, mixed methods and boundary-spanning work.

Future

As AI advances, we expect that researchers will work at higher levels of abstraction. Now we design research instruments; tomorrow we'll propose research questions and the AI will propose surveys; next week the AI will be suggesting what research questions we should even be asking and doing all the associated research. Our tools will automatically construct the right agents, run the right simulations, do the right analyses and present the key results—perhaps all within seconds. And as our knowledge of how to combine humans and AI answers improves (an active area of research), we'll get precise, automatically generated predictive models that optimally make use of both sources of information. One could think of it as an infinitely scrolling map (but flawed, like all maps) that gets built on the fly so quickly that it seems like the full map already exists. It's going to be a wild world and we're excited about helping to build it.

• EDSL, an open-source Python library for simulating surveys and experiments with AI agents and LLMs (read our docs); and

• Coop, an integrated platform for conducting hybrid AI/human research interactively (login/signup).

The team had heard about our tools and thought they could be helpful for gathering information about potential customers for startup ideas. But the presentation went sideways when I brought up a slide of EDSL code (and accompanying snapshot of our interactive survey builder tool at Coop) to show how to begin a research experiment-as-a-survey for LLMs:

The same first step at Coop (you can click “Code” to get the EDSL code, for technical and non-technical teams to collaborate):

The team was insistent that early-stage founders, particularly non-technical ones, don’t have time to learn a new tool for time-sensitive market research. In fairness, you can obviously get a ton of fast research done with user-friendly AI chat interfaces. But if you want to do any pressure testing of the answers that you’re getting from LLMs—What happens when I emphasize this part of the customer persona instead? What happens when I modify the description of the product, or rephrase the question, or try a different model, or provide more information? etc.—it quickly becomes unwieldy to collect data and analyze feedback and ideas via unstructured text.

It also makes it hard to compare your AI results to any information that you’re able to collect from your actual target audience. In order to compare your LLM and human answers you’ll want a dataset—even a simple spreadsheet—with organized information about the personas that you included in your LLM prompts, the information that you have about your human respondents, and the answers that each of them provided to your questions. Your dataset should also show all the variations of the prompts that you sent to LLMs, which is only possible if you…

Tinker with your prompts!

It’s not necessarily obvious what information you have—or do not have—about your target audience is relevant to the questions that you want to ask, or how its presentation in LLM prompts is impacting the responses. If you want to increase your confidence in your AI results you should tinker with your questions and AI personas and include humans in-the-loop when possible. EDSL is designed to make all of these things easy in a number of ways:

• Explore question types—EDSL has many common question types (free text, multiple choice, etc.) to explore the impact of the presentation of a question on responses.

• Test variants of your prompts—EDSL has many methods for auto-importing data to efficiently run multiple versions of your questions, instructions and agent personas all at once.

• Compare models—EDSL works with many popular service providers, and your account comes with a key for accessing all of them at once to easily compare performance on your tasks.

• Validate with humans—EDSL provides methods for launching your surveys with humans to compare results and improve your workflows. Choose whether to send surveys to your own respondents or use our Prolific integration.

• Analytical tools—Results—LLM and human—are returned as formatted datasets containing details about questions, agents, models, responses and costs that you can immediately begin analyzing with build-in methods, or export.

EDSL is easy to learn.

No, you do not need to be any kind of coding expert or Python wiz to master EDSL. Our docs have demo notebooks for a variety of use cases that you can download and modify to get started. You can also use Coop to create a project interactively and then get the EDSL code for it (click the “Code” button in the snapshot above), or use our GPT for generating EDSL code. The concept—design an experiment as a survey answered by AI personas using LLMs to generate the answers—and the base components are straightforward:

• Construct questions (free text, multiple choice, linear scale, matrix, etc.)

• Combine them in a survey and add any desired logic (e.g., skip/stop rules)

• Optionally design personas for AI agents to answer the questions

• Send the survey and agent personas to a language model to generate the responses

• Get the results back in a formatted dataset to analyze with built-in methods

Here’s a notebook with a quick example. I start by constructing a survey and AI persona and select a model to generate the responses:

Next I launch a web version of the survey answer it (at the respondent URL):

I can pull the results back into my workspace to compare with the LLM results:

… and also inspect them at my Coop account:

Market research & other use cases

If I can convince you to give it a try, here are some ways to use EDSL to conduct market research:

• Simulate customer personas: Design Agent objects with a variety of traits and preferences to represent different customer segments.

• Test messaging and branding: Use question types such as QuestionFreeText or QuestionMultipleChoice to get feedback on product descriptions or branding materials.

• Compare product features: Create Scenarios for different feature sets and use QuestionRank to have the agents rank preferences or interest.

• Evaluate price sensitivity: Present different pricing scenarios and ask agents to choose or rate affordability using QuestionLinearScale.

• Analyze perceived value: Use open-ended questions to elicit reasons behind preferences or concerns with QuestionFreeText.

• Explore distribution channels: Pose Scenarios about online vs. in-person availability and ask which they prefer and why.

• Gauge emotional response: Include qualitative questions with QuestionFreeText to measure reactions to marketing copy, visual content, or product pitches.

• Test variations across contexts: Vary Scenario and Agent attributes to explore how preferences change by traits or product conditions.

• Benchmark against competitors: Create Surveys comparing your product with existing ones and assess perceived strengths and weaknesses.

If you have any questions about using our tools for your research, please get in touch! info@expectedparrot.com

1. Create & run a survey in EDSL

Start by constructing questions in EDSL, our open-source package for running surveys and experiments with AI agents and LLMs. Choose from many common question types based on the form of the response that you want to get back from a model. You can optionally design personas for AI agents to answer the questions, and specify which of many popular LLMs you want to use to generate the responses. Run the survey by adding the agents and models to the survey and calling the run() method. This generates a formatted dataset of results that you can analyze with built-in methods:

When you run your survey at Expected Parrot your results are automatically stored at your Coop account where you can access and share them. You can check a progress report while the survey is running, and see details on costs for each response when it is done:

2. Inspect results

Results include information each component of the survey job: questions, prompts, agents, models, raw responses, costs, etc. You can inspect results at your Coop account, and also use methods for analyzing them at your workspace. Here we select the responses, and the comments that are automatically added to them (learn more about modifying user and system prompts for your specific research needs):

3. Generate a web-based version of the survey

To collect actual human responses to your survey, call the humanize() method on the Survey object to generate a web-based version of it. You’ll get a link that can be shared with anyone you want to answer the survey, and another link for accessing responses at your Coop account (the admin_url):

You can also use our interactive survey builder tool to construct a different web-based survey, or to edit the one you generated from your EDSL survey (e.g., if you want to add some screener questions for your real respondents).

4. View & share the survey URL

Here I go ahead and answer the survey myself, as my response can offer a reliable check on the AI agent’s from above:

No problem!

5. Combine & analyze results

LLM and human results are datasets that you can analyze with built-in methods for working with results (e.g., to use them as inputs for follow-on questions for LLMs). You can also export them, e.g., as dataframes:

We’re always adding new methods like these — please send us your feature requests! Send us an email (info@expectedparrot.com), post a message at our Discord or DM us on X.

Below we share some quick examples of methods for retrieving multiple sets of results at once, combining them, and analyzing them with SQL queries. This can be useful when you are re-running an experiment with different AI agents and/or language models, or want to compare fresh responses over time.

Retrieving saved results

When you run an EDSL survey, details about the survey job and results are displayed in your workspace while the job is running (you can also hide this output by passing verbose=False):

When the job is completed the results are automatically stored at your Coop account, where you can view, organize, share and export them (a link to Results is in the job status details shown above):

You can retrieve stored results at any time by passing the unique identifier to the pull method:

You can also give the results a unique alias, if desired:

This allows you to retrieve by alias instead:

Multiple results

If you want to retrieve several sets of results at once you can use the list method to identify and select them. For example, say you reran the survey above multiple times with different models:

You can then specify which results to retrieve, e.g., the 3 most recent:

The page specifies which set of 10 objects to look back to (1 by default, the most recent page) and page_size specifies how many objects to return (10 by default). If you want to search results by survey job status or description instead you can call the method on Jobs (all parameters are optional):

Once you’ve identified the relevant jobs or results you can use the fetch methods for instantiating the results. These are equivalent:

This returns a list of results objects. You can combine them to analyze them together (the columns must be identical):

Methods for analyzing results

Now all of the methods for working with results are available to us. For example, you can use the sql method to run queries on the combined results:

If you want to share an analysis, or a notebook of code for reproducing it, you can post that to Coop as well:

Please see our docs for more examples of analytical methods and instructions for getting started. If you don’t see a method that you need, please post a message at our Discord channel or send us an email to request it.

Question: How many Turing Award winners graduated from MIT?

Say I want to know how many Turing Award winners received a degree from MIT. This is a very answerable question, but there very likely isn’t an exact dataset I need with an “mit_degree” boolean field. I’ll need to construct it.

In the distant past (say, 3 years ago), I could have taken Wikipedia bios and maybe done a regular expression search for “MIT”—though given how many winners are affiliated with MIT but did not graduate, this would be error-prone. Ultimately, I’d probably have to review by hand. Now, of course, we can use AI.

First, I’ll show approaches that don’t really work (with Claude) and that do work (with o3) but also a method that works with less sophisticated models and that has a number of other advantages as an answering approach.

Claude gives an MIT ‘vibes’ answer

The Claude answer definitely finds some related info but includes people with MIT connections but not actual degrees. This is probably because every time someone wins a big award, universities try to establish some connection (“Herb Simon once ate in our cafeteria!”). There is also a counting effort, but garbage-in, garbage out.

o3 seemingly nails it

I asked OpenAI’s o3 “How many Turing award winners received their PhD from MIT?” and then asked about degrees generally. Watching the reasoning traces as they flashed on the screen, it seemed to do something sensible and probably what one of us would do if given the task—it got a list and then kept doing web searches until it found the relevant information, almost all from Wikipedia:

As far as I can tell, this is correct. But this was PhDs—let me instead do graduate degrees, which is what I mean to ask:

Ok, 6.

But even o3’s answer has some problems

The procedure with which this answer was obtained is opaque.

• I don’t really know the process by which these determinations were made.

• I don’t know if any of these are hallucinations (and I can’t easily try it with other models) or omissions.

• If someone wanted to reproduce my analysis, they would have to re-run this (rather expensive) inference (2m 22s).

And what if my dataset were *a lot* bigger and not publicly accessible. There are only a fairly small number of Turing winners and the list can be found in numerous places. What if I want to do something similar but with non-public data of much larger scale—say look for who graduated from MIT from my 10,000 employees in a CSV sheet?

The E[🦜] Approach

First, the data on winners is easy to get, already arranged in a nice table at Wikipedia:

To actually extract this, EDSL (Expected Parrot Domain Specific Language) has some tools for working with Wikipedia data. The basic idea is turn it into a “ScenarioList” which is, well, a list of Scenarios for a language model to assess in some way.

The data is puled down using “from_source” method. EDSL uses a fluent interface throughout and so we can chain together methods. In particular, the keys need to be valid python identifiers, which is what that “give_valid_names()” function does. There is also an “augment_with_wikipedia” function that can get get more data about a particular search query—-in this case, the prize winner.

Here is what one entry looks like:

Note that I have a field that has the full biography from Wikipedia. I can then use that to ask a question: Did this person receive a degree from MIT? For this, we use a yes-no question type. Note the use of a jinja2 templating syntax:

The last line uses a fluent interface to combine this question with the scenario list and then “run” it with a language model. (The current default model in EDSL is gpt-4o; you can easily specify a different model, or run it with several models at once to compare responses.) We can see that answering question takes about 3 cents’ worth of tokens:

If I—or anyone else—were to run it again, the responses would be returned at no cost, as they have been “cached” at E[P]. With our results object, we can now tally up the results:

Oh, wow! But wait—boy genius o3 is wrong!

Note that o3 said 6, but the EDSL approach got us 7. NB: This wasn’t a big set-up to show our tools work better. I just noticed this as I was working it out. What’s going on? Forgetting someone?

Turning the knife:

You can view this and other metrics for the Universal Remote Cache at our cache stats page:

This page is updated daily to show total cache entries and total savings (costs avoided whenever a stored response was retrieved instead of resending an API call). The information is available for each language model used, so you can also tell at a glance which models are popular for research projects.

What is the Universal Remote Cache?

The Universal Remote Cache (“URC”) is a collection of all the unique prompts (user prompts and system prompts) that have been sent to language models via Expected Parrot, together with all the responses that were returned. It is a shared resource that is available to all users for free whenever you run a survey at Expected Parrot.

How it works

When you run a survey at Expected Parrot your results draw from the URC by default. This means that if your survey includes any prompts that have been run before—by you or another user—the stored responses to those prompts are retrieved from the URC and included in your results, at no cost to you. A new response is generated for each of your prompts that has not been run before, and added to the URC. If you want to generate all fresh responses instead you can turn this default off when you run your survey.

Your prompts and responses are shown at the Cache page of your account; details on costs (and costs avoided) in generating them are available at your Jobs and Transactions pages. To show how it works, below we run a simple question with some models and check the job, transactions and cache details. Then we re-run it and verify that the cached responses were retrieved and no credits were used.

Running a survey in notebook:

The survey job is added to the Jobs page with information about credits used to generate the results:

When we re-run the question we can see that no credits were consumed for the second job:

At the Cache page we can verify that 3 entries were created and then retrieved:

We can generate fresh responses by passing a parameter fresh=True to the run() method:

We can see that credits were consumed again to generate the new responses:

The cache has new entries (fresh unique responses to existing prompts are automatically indexed):

How remote inference works

This caching of prompts and responses happens automatically whenever you run a survey using EDSL. The URC is used whenever you run a survey “remotely” at the Expected Parrot server; when you run a survey “locally” on your own computer your prompts and responses are cached on your computer instead.

To activate remote inference, navigate to the Settings page of your account and confirm that the remote inference option is toggled on. This allows you to run your surveys at the Expected Parrot server using your own API keys for LLMs and/or an Expected Parrot key. Learn more about how remote inference works.

Features

Features of the Universal Remote Cache include:

• Free access: The URC is free to use and available to all users, regardless of whether you are running surveys at Expected Parrot with your own keys for language models or an Expected Parrot key.

• Free storage & retrieval: There is no limit on the number of responses that you can add to the URC or retrieve from it.

• Automatic updates: The URC is automatically updated whenever a survey is run at Expected Parrot.

• Multiple responses: If a fresh response is generated, it is automatically added to the URC as well, with an iteration index.

• No deletions: You cannot delete entries from the URC.

• Only verified responses: You cannot manually add entries to the URC. The only way to add responses is by running a survey at the Expected Parrot server.

• Privacy: The URC is not queryable, and no user information is available.

• Sharing & reproducibility: A cache for a survey is automatically attached to each results object, which can be shared with other users. A green checkmark at Coop indicates to other users that your results are verified and can be retrieved from the URC for free:

  

  Note: The URC is not available for local inference (surveys run on your own machine).

Frequently asked questions

How do I add responses to the URC? This happens automatically when you run a survey at the Expected Parrot server.

How do I retrieve a stored response? When you run a question you will retrieve a stored response by default if it exists. If you want to generate a fresh response you can use run(fresh=True).

How do I know whether I will retrieve a stored response? Results that were generated at the Expected Parrot server will show a verified checkmark ✓ at Coop. If you rerun a survey with verified results you will retrieve stored responses by default.

Is the URC queryable? Can I check whether there is a stored response for a question? No, the URC is not queryable. You will only know that a response is available by rerunning surveys with verified results.

Can I see which user generated a stored response in the URC? No, there is no user information in the URC.

Can I add my local cache to the URC? No, this is not allowed. The purpose of the URC is to provide a canonical, verified collection of responses to allow researchers to be confident in results and easily reproduce them at no cost. By only allowing responses generated at the Expected Parrot server we can verify the results that are reproduced.

What if I want to run a survey remotely but do not want my responses added to the URC? This is not allowed. Any new or fresh responses generated at the Expected Parrot server are automatically added to the URC. If you do not want to add responses to the URC you must run your surveys locally.

Can I access the URC when I run a survey locally? No, the URC is only available when running a survey remotely. However, you can pull entries from your own cache (responses that you generated or retrieved from the URC) to use them locally at any time.

Can I delete a response in the URC? No, this is not allowed.

What happens if I delete my account? Any remote cache entries that you generated will remain in the URC. All other account information will be deleted.

Please send us other questions about how it works!

When you run a research project with Expected Parrot it’s easy to track the details of your LLM costs and usage, using built-in methods at your workspace and at your account at Coop, a platform for creating and sharing LLM-based research. You can also replicate and share your results for free. Below we highlight some new and key features for easily and precisely tracking your costs.

Before you run an experiment…

When you’re developing an LLM research project you want to know (i) the token rates for the models that you plan to use and (ii) estimated costs for your prompts and the responses that you will receive back from the models. You also want to avoid making duplicative API calls to LLMs (resending identical prompts when you do not expect the response to change) to eliminate unnecessary $$ charges.

Checking model prices & capabilities

It can be time-consuming to gather token rates from various model service providers’ webpages, and to determine which models are capable and appropriate for your research goals and requirements (e.g., do you need a vision model for some of your prompts?).

This is why we created a model pricing and performance page where you can quickly check current token rates for models of all available service providers, and their recent performance on test questions:

The page is updated daily with details on the test questions that were used with each model. We are adding test questions that use videos! See example code for using videos with your questions.

Estimating costs

Once you’ve drafted your user and system prompts (i.e., created your questions and any agents), you can use built-in methods for estimating the costs of sending them to models (input tokens) and the costs of the responses that you will get back from the models (output tokens).

For example, here we create a simple question and then use a method for estimating input and output token costs of sending it to two different models. We can see the details also include the total credits that will be placed on hold while the survey job is running and then released when the actual cost is determined:

After a survey is run, we can verify the actual token costs for each model and question in the dataset of results that is generated:

You can also view token and cost details at your Coop account. Your Jobs page shows the final cost in credits of API calls to each model used with the survey:

Your (new!) Transactions page shows additional details about credits placed on hold based on job cost estimates together with the final cost details:

Avoiding unnecessary API calls

A key feature of EDSL is automated caching of prompts sent to models and responses received from them. This ability to retrieve responses to questions that have previously been run at no cost means that you can replicate and share any research created in EDSL for free.

For example, if we rerun the example question from above we can confirm that no additional costs are incurred:

This can be useful when iterating on a research project and when publishing your research results for others to review and replicate.

• Unified access: Connect to multiple language model providers with a single API key and universal methods—no software engineering required

• Structured data collection: Automatically formatted datasets of responses eliminate manual data cleaning work

• Analysis tools: Built-in methods for analyzing, reproducing and exporting results

How it works

EDSL is an open-source Python package designed to simplify conducting experiments with AI agents and language models. A typical workflow consists of the following steps:

• Create questions (free text, multiple choice, numerical, matrix, etc.)

• Combine questions in surveys with custom logic (skip patterns, stop rules, etc.)

• (Optional) Create personas for AI agents to answer the survey

• Select language models to generate the responses

• Analyze results in specified datasets

Simplified access to models

EDSL is designed for researchers who want to work with language models without spending lots of time on software development. It eliminates the need for one-off coding required to access individual service providers by letting you connect to them all in the same straightforward way.

In EDSL, you can access models individually or collectively by simply selecting them and specifying desired parameters. For example, here we create a list of OpenAI models in order to use them all simultaneously with a survey when we run it. We can inspect the default parameters for the models that will be used:

You can also create multiple instances of the same model with different parameters. Here we create a new model list where we adjust the temperature of a single model:

Results as formatted datasets

Running a survey with multiple models at once allows us to readily compare responses in the dataset of results that is generated. Here we run a simple survey (consisting of a linear scale question and a numerical question—see examples of all question types) and inspect the models’ responses:

These results can be viewed at Coop as well:

Other columns of the results include:

• User and system prompts

• Question details

• Agent traits and instructions

• Model temperature and other settings

• Raw and formatted responses

• Log probs

• Input and output tokens

• Costs

Works with many models

EDSL works with many other popular language models as well. You can check available models, current token prices and daily performance on test questions here:

Please get in touch to request other service providers that you want to use!

Getting started

Our documentation page includes tutorials and demo notebooks for getting started using EDSL with language models, including examples of methods for evaluating model responses.

If you have a use case you don’t see, please see us a message and we’ll create a notebook for you!

1. Open source features. Our platform is built on EDSL, our open source package for conducting research with AI agents and language models.

2. Designed for researchers. Our tools are designed for social science and market researchers who want to conduct complex tasks and experiments with LLMs—and not spend lots of time on software development.

3. Easy access to models. Our tools work with many LLMs from popular service providers that you can connect to with your own API keys or a single key managed by Expected Parrot. You can check current performance and token prices of available models here.

4. Reproducibility. Your LLM prompts and responses are stored automatically, and always available to you. Anyone else can also retrieve them for free whenever you share your experiments and results.

5. Transparency. Our tools allow you to precisely specify and control all components of your experiments. Results are generated as readily usable, formatted datasets.

6. Built-in tools for analysis. Our tools come with built-in methods for analyzing, visualizing and exporting your results.

7. Free storage. Your account comes with free storage for your projects so you can access them from anywhere.

8. Free sharing. You can choose whether to share projects and results privately, or post them publicly.

9. Free credits for API calls. Your account comes with $25 in credits for API calls to language models for getting started. We also have lots of free credits for students and research projects!

How it works

Expected Parrot tools are designed to make it easy to run complex experiments with LLMs, and to share and replicate research. A typical workflow consists of the following steps:

• Design an experiment in EDSL, our open source Python package.

• Run your EDSL code in a notebook (e.g., Jupyter, VS Code or Colab).

• View your results at your account (Coop is our free platform for storing and sharing content you create in EDSL).

• Iterate on your work using built-in methods for analysis.

• Publish your work to allow others to replicate your research at no cost.

Technical setup

Technical setup is quick!

1. Create an account with an email address.

2. Install the EDSL package on your computer. See instructions.

Explore demo notebooks

If your experiment idea is nascent, take a look at our many demo notebooks of example code for a variety of use cases. These are designed to help you get familiar with how EDSL works and also provide a modifiable template as your starting point.

Try our GPT

Not sure how your idea would work in EDSL? Try describing it to our GPT for generating EDSL code. Here’s a quick example:

Copy the code to a notebook and run it:

You can view the results at Coop:

Please get in touch if you have any questions or feature requests!

Post a message at our Discord or send us an email at info@expectedparrot.com.

Choosing where to run your surveys

EDSL is an open-source Python package that lets you conduct surveys with language models either “locally” on your own computer or “remotely” at the Expected Parrot server.

Getting started

The first steps are the same for either option:

1. Install the EDSL package. See instructions.

2. Create a survey. See our starter tutorial and other examples. You can also use our GPT for generating EDSL code.

Then decide whether you want to run your survey locally or remotely.

Activate remote services

To active remote services, start by logging into your account and navigating to your Settings page. Remote inference is activated by default when you create your account:

While the toggle is on, you can run surveys remotely simply by calling the run() method on a survey at your workspace:

A table of information about the status of the survey job will appear while it is running, with a link to a progress report. When the job is completed, a link to view the results at Coop will appear as well:

You can also access and analyze the results directly in your workspace.

To run surveys locally instead you can either turn the toggle off or pass a parameter disable_remote_inference=True to the run() method:

What happens when you use remote or local inference

When you run a survey locally, your prompts are sent to language models from your computer and responses are delivered back to your computer.

When you run a survey remotely, your prompts are sent from and responses are delivered back to the Expected Parrot server. Details on the survey job and results are automatically added to your Coop account, where you can choose whether to share them and how you want to work with them. Prompts and responses are also added to the universal remote cache. This means that if you re-run any prompts in the survey, or share the survey with anyone else, you will have the option to retrieve the stored responses at no cost. You can also specify that you want to generate fresh responses instead.

Choosing how to run your surveys

EDSL works with many popular language models you can choose from to generate responses to your surveys, including models from Anthropic, Azure, Bedrock, Deep Infra, DeepSeek, Google, Groq, Mistral, Ollama, OpenAI, Perplexity, Together and Xai. You can check available models, token prices and daily performance on test questions here:

When you run a survey remotely, you can connect to language models using your own API keys from service providers or using a key from Expected Parrot that provides access to all available models.

Note: You must provide your own keys when running surveys locally.

How it works

Your Expected Parrot key is accessible at your Keys page, where you can reset it at any time:

If you have other keys that you want to use when running surveys remotely, you can select the “Add key” button to add them. You can deactivate and delete them at any time:

You can also choose whether to grant access to your keys to other users (without sharing keys directly) and set usage limits.

If you are running surveys locally, you must store your keys on your computer. We recommend doing this using a .env file. See instructions on managing your keys.

Your Expected Parrot key also provides access to Coop

Your Expected Parrot key also lets you connect to Coop: an integrated platform for creating and sharing AI-based research:

Your account lets you post, download and share content, including surveys, agents, results and notebooks. When you run surveys remotely, results and survey job details are automatically available at your Coop account, where you can choose whether to publish and share them privately with team members or publicly. Results that are generated remotely will also show a verified ✔️ checkmark for reproducibility.

You can also post any other content from your workspace to Coop, whether created locally or remotely. Learn more about methods for working with Coop.

Whenever you run an experiment, you want to write down your steps in such a way that someone could replicate your work without much effort. When you’re working with LLMs, this means showing—and therefore also knowing—exactly what prompts were sent to which model. That sounds pretty obvious and comparatively easy to pull off when your experiment is essentially a bunch of computer code that you can simply post on the internet.

But, alas! Models get replaced, requirements for connecting to them change, and it costs money to resend API calls to them. Even if someone eagerly wants to reproduce your work, they may by stymied by tier limits when creating their own accounts with relevant service providers.

Add to this the fact that, depending on the tools used to run your experiment, the precise details of your prompts and API calls may be difficult or impossible to ascertain, making it hard to know whether you are actually repeating your steps.

These issues may render your experimental code a bit of a relic. What’s needed to help prevent this are:

• Accessible copies of your prompts and responses, so anyone can work with them

• Evidence of the way your responses were generated (which model, what time, etc.), especially if they can no longer be regenerated

• Code that is easily—or automatically—updated to work with current models and requirements, and that can be extended to facilitate follow-on work

These to-do’s are all straightforward to accomplish using EDSL, our open-source Python package for simulating surveys and experiments with LLMs, together with Coop, our integrated platform for creating, sharing and replicating LLM-based research. Below I show some features of EDSL and Coop for crossing each of the above tasks off the list. Please send us questions about any of them!

EDSL results are standardized datasets

When you run a survey in EDSL, the LLM responses are returned to you in a formatted dataset of results that is specified by the question types that you used (free text, multiple choice, numerical, matrix, etc.). The results automatically include (many!) columns of information about all the details of the survey, including:

• Question texts and types

• Agents and data used with the survey

• User and system prompts

• Model names, temperatures and other parameters

• Survey rules and logic applied

• Raw and formatted answers

• Tokens consumed

• Costs incurred

• Timestamps

• and more

There is no need to perform any post-survey data cleaning to extract this information from raw responses—it is all done automatically.

Share verified results privately or publicly at Coop

If you run your survey at the Expected Parrot server, the results are automatically added to your Coop account, where you can share them privately or publicly, or simply store them for free. A green checkmark at your dashboard verifies to anyone with access to your results that they were generated in this way:

The prompts and responses are also automatically added to your own cache and a universal remote cache, so that anyone who reruns your survey can retrieve them—at no cost.

Rerun your code with any available models

EDSL works with many popular language models. An important feature of the package is that you can pick and choose models to use with your surveys without having to do any of the high-touch software dev work needed to make API calls to various service providers. It works the same way with all available models.

For example, to run a survey with models from Anthropic, Google and OpenAI all at once we simply add them to the survey by name in the exact same way:

The results that are generated will have separate rows for each set of responses by each model, with information about each model (temperature, etc., which you can specify—in the example above we use the default settings).

Learn more about working with models.

Tinkering with prompts—how do I do that?

User and system prompts are determined by (i) the contents of your questions, (ii) details of any agents you use with your survey (they are optional), and (iii) instructions for models about the questions and agents. EDSL comes with built-in methods for editing and inspecting these instructions before you launch a survey. After a survey is run, the user and system prompts for each response are stored in columns of the results.

For example, here we show the user and system prompts that are generated for a simple question and agent with default instructions, and then show how they change when we add parameters for modifying the instructions:

We can see that the default instructions for the question include (i) an instruction about how to answer the question and (ii) and an instruction to provide a comment about the answer. This instruction to provide a comment is automatically added to all question types other than free text, and can be useful for debugging issues with answers.

We can edit or omit these question instructions by specifying either or both of the optional parameters answering_instructions and question_presentation when we construct the question. Similarly, we can edit or omit the instructions in the system prompt (“You are answering questions as if you were a human. Do not break character.”) by passing an optional parameter instruction to the agent constructor. Here we demonstrate all of them at once:

Here we run the surveys with the agents to compare the responses (the results are also accessible at Coop here and here):

Code for the above examples is available to download and modify for your own purposes in this notebook at Coop.

Learn more about methods for working with prompts in our docs.

But I don’t know Python!

Several pieces of good news here:

1. EDSL is designed to be intuitive to anyone wanting to conduct this kind of question-and-answer LLM-based research. Advanced Python or other programming language skills are absolutely not required.

2. We have lots of tutorials and demo notebooks that you can download and modify for your own purposes, and we’re available for 1-1 help at many hours of the day.

3. You can use our GPT to generate EDSL code too.

   

4. Our no-code survey builder interface lets you launch surveys with AI agents and send web-based versions to human respondents. You can compare and export responses from your Coop dashboard, and get the underlying EDSL code to iterate on it in a notebook or share it with your team. Send us an email if you’d like a quick demo (see a quick video and read how it works):

   

5. We have special features for automatically converting your existing surveys (Qualtrics, CSV, etc.) into EDSL. Send us an email to request access.

Please let us know if you need any help getting started! Feel free to post a question at our Discord or send us an email: info@expectedparrot.com

There are several ways to explore this in EDSL, our open-source Python package for simulating surveys and experiments with AI agents and large language models.

How to explore this in EDSL

EDSL has built-in methods for adding context of questions and answers to other questions within a survey or experiment, giving you precise control over the information that is included in each prompt. These include:

1. Question types that prompt a model to return responses to multiple questions at once in a dictionary or list.

2. Methods for piping specific components of questions into other questions—e.g., to insert just the answer to a question in a follow-on question.

3. Methods for adding the full context of one or more prior questions and answers to the presentation of later questions.

A one-shot multi-part question might work

A simple-sounding way to give an agent full context of a survey is to include all the questions at once in a single prompt, together with instructions on how to answer them. With longer sets of detailed questions, however, this can lead to incomplete and inconsistent results. Even if a model’s context window has not been maxed out, a model may be distracted by the amount of content presented and fail to precisely follow each of many instructions presented at once. If multiple agents are answering a survey, the responses may also not be consistent among the agents, and require post-survey data cleaning.

Use a structured question

A better way to present multiple questions at once is to use a structured prompt—a presentation of the questions in such a way that, when administered multiple times to different agents, will ensure consistently formatted results to facilitate analysis.

In EDSL, the question type that you choose—free text, multiple choice, numerical, etc.—determines the format of the dataset of results that is generated when a survey is sent to a model. If you want to ask multiple questions at once, a convenient question type to use may be QuestionDict or QuestionList. For example, here we present the same set of questions in both formats:

When we administer the questions to an agent and model we get the multiple responses back in the specified formats, ready for export and analysis:

We can inspect and export the results at Coop too:

Code to reproduce this example is also available in this notebook at Coop.

Piping answers into other questions

Another tool that allows you to precisely control the information that is included in each prompt while keeping prompts efficient is piping. Instead of piling entire questions together, you can specify exactly which parts of prior questions and answers you want to include in later questions. For example, here we just insert the response to q1 into the context of q2, allowing us to keep the context of q2 from becoming unnecessarily long:

We can also access these results at Coop:

Code to reproduce this example is also available in this notebook at Coop.

Adding context of specific prior questions

This can be especially useful when your survey has independent sub-sections of questions that only require the context of other questions in the same sub-section, or when you want to explore the impact of particular context. For example, here we create a survey of 3 questions and add a survey rule to present the context of q1 in q2, but not q3. We inspect the prompts to verify what will be sent to the model (answers are ‘None’ because the survey has not been run yet):

Adding full context of prior questions and answers

We can also add a survey rule to present the full context of all prior questions and answers to later questions to each consecutive question. For example, here we revise the above survey to present the full context of all prior questions at each subsequent question:

Note, however, that this method can quickly produce very long question prompts as the set of questions grows.

Modifying prompts and instructions

In an upcoming post I’ll discuss and demonstrate built-in methods for modifying question type prompts and default instructions. These methods can also be useful in exploring and experimenting with agent memory!

When asked what Expected Parrot does, I usually say:

I add “experiments” because “surveys” does sound niche, and I want to convey that there are many things you can do besides simulating traditional surveys. I mention “AI agents” because EDSL lets you design personas for language models to use in answering your surveys. But there are lots of use cases where AI agents are not needed at all.

So what are these other use cases?

Below are quick overviews of some of these other use cases. Really, any time that it would be helpful to have a language model answer some question and get the response back in a formatted dataset, you can probably use EDSL.

If you have a use case that you don’t think squares with any of these, or my explanation is lacking, please let me know! I love creating demo notebooks.

Simulating games & other agent interactions

You can simulate interactions between agents as a series of one-question surveys sent back and forth between agents, with the context of prior questions and answers automatically included at each step. You can also do this with multiple agents to simulate rounds of games with complicated instructions and information about other agents’ choices/actions. Our conversation module provides built-in methods for setting this up with example code for interactions and negotiations:

Join our Discord to connect with researchers who are using EDSL for simulating games and can share more examples!

Qualitative analysis via free text responses

When conducting exploratory research on complex topics, free text questions allow for rich, nuanced responses. For example, we could use a model to review and comment on some content (note that we can use a “scenario” placeholder in the question in order to re-use it with different content):

After generating qualitative responses, we can also use EDSL to extract and quantify themes that appear in the responses.

Quantifying qualitative data

One way to do this:

1. Use a free text question prompting a model to provide some qualitative content.

2. Then run a question prompting a model to review each free text response and identify the themes in it (see data labeling examples above).

3. Then run a question prompting a model to turn the set of all themes into a non-duplicative summary list of themes (i.e., create a new short list that covers all the individually-identified themes).

4. Use the summary list as options for a checkbox question prompting a model to review each free text response (again) and select all the themes that appear in it.

5. Use built-in methods for quantifying the checkbox results.

See an example of this (quantifying course evaluations).

We also have a new module for automatically extracting and quantifying themes in qualitative content. Please send me a message to request access!

Data labeling & cleaning

I recently wrote about this use case which entails designing surveys about your data, where you use a language model to answer questions about each data point in a dataset. For example, we could choose a different question type for the above question and use it for a large collection of texts all at once:

EDSL will return the responses from the model in a formatted dataset that is specified by the types of questions that you use—free text, linear scale, numerical, matrix, etc. You can also use multiple models at once to compare performance (see the next example below), and use methods for auto-importing data from many different sources (images, docs, CSVs, PDFs, tables, etc.).

We have lots of notebooks for this use case that you can download and modify for your own content and purposes.

Comparing model performance

If you’re doing research on language models themselves (as opposed to using them to assist your research, such as with data labeling tasks), you can use EDSL to simplify the task of gathering data on multiple models at once in a convenient dataset.

EDSL works with many popular service providers, including Anthropic, Azure, Bedrock, Deep Infra, DeepSeek, Google, Groq, Mistral, Ollama, OpenAI, Perplexity, Together and Xai. You can check model availability, token rates and performance on test questions here:

To collect responses from multiple models at once we simply add them all to a survey when we run it:

This produces a dataset of results containing responses for each model that was used, with columns for all of the components of the survey job, including the model parameters that you can specify. If you don’t specify a model to use with a survey the default model is used; you can inspect the default params:

When you add models to a survey you can choose whether to use your own API keys for models or use an Expected Parrot key to access all models at once. You can also mix and match keys, share access with team members and set usage limits (learn more):

You can simulate traditional surveys too

Finally, here’s a use case that is most likely envisioned by my description of the tools:

For example, here’s a quick survey where I am personally qualified to report that the model’s response is pretty accurate:

I’ve shared the results publicly, so anyone can view them at Coop, replicate the results for free, and re-use the survey however you want.

Launch hybrid human/AI surveys

If I want to add human responses to the mix (or I don’t want to write code), our no-code Survey Builder interface lets you design and run EDSL surveys with AI agents and send a web-based version of the survey to human respondents. Here I recreate the above survey:

I’ve also made the Coop page public, so anyone can view and re-use it.

Feel free to add your response!

Whether or not your ultimate task or research goal has anything to do with language models, using a model to perform data labeling tasks can be very helpful to your process.

How does it work?

Using a language model to perform a data labeling task typically consists of:

1. Sending your data to a model

2. Giving the model instructions on how to label (analyze, restructure, etc.) the data

3. Getting back labeled data you can use

Benefits of using a language model

If you can get it working, this method can provide you:

• Substantial cost & time savings. Many data labeling tasks require a lot of human effort that may not be available. Using a model can not only save you time and money, it can unlock new projects that you wouldn’t have been able to complete without automated assistance.

• Superhuman consistency of results. A model will tirelessly work through your large datasets!

• Reproducibility. You can build—and iterate on—processes that are easily reproducible. And the task gets easier to set up as models get better.

Some do’s and don’ts

It can be tempting to try to do it all in one go. Context windows are big; if your dataset fits, you can always write a smart prompt, send all your data and hope for the best. You might also want to ask all your questions at once to save on tokens.

But you may have better luck if you break the task into steps. Rather than prompting a model review your entire dataset at once, instructing a model to evaluate data individually, and one question at a time, may produce more consistent and complete results.

Challenges of step-wise labeling

Challenges of this approach—evaluating a dataset piece by piece instead of collectively—include figuring out how to avoid or automate repetitive steps and collecting all the outputs (individual labeled data and responses) in an organized way.

This is where we think EDSL can provide a lot of value.

Why EDSL

EDSL is an open-source Python package for simulating surveys with AI agents and language models. It is designed for researchers who want to conduct in-depth experiments with many agents and models and easily replicate results, while avoiding one-off software development work required to connect with different service providers.

A key feature of EDSL is that survey results are returned to you as specified datasets, where the format is determined by the question types that you use—free text, multiple choice, linear scale, matrix, dictionary, etc. This makes data labeling an ideal use case, where you create questions about your individual data and get back a formatted dataset that is determined by your question types and instructions. In a nutshell:

Design the task as a survey about your data

Here’s a quick example of how it works (you can view and download it in a notebook):

1. Import your data. EDSL provides methods for auto-importing data from many different sources (CSV, PNG, PDF, tables, texts, etc.) and creating “scenarios” for the data in order to add it to questions. Here we create a scenario for a Japanese print and provide a dictionary key (“image”) that we’ll use to add the print to questions:

   

   

2. Construct questions about your data. We can choose from many common question types based on the format of the response that we want to get back from the model (free text, multiple choice, linear scale, matrix, etc.). Here we construct some simple questions about the print, using a placeholder for the key that we specified:

   

3. Combine the questions in a survey. We can also add skip/stop rules and other logic to conduct multi-step processes:

   

4. Send the survey to a language model to generate the responses. We can also use multiple models at once to compare responses in the same dataset of results:

   

5. Get a formatted dataset of results. We can access the results at Coop:

   

   
   Or use built-in methods for analyzing and exporting results from your workspace:

   

   

You can view and download this example in a notebook. The code is readily modifiable for your own content and purposes.

More examples

Here are some more demo notebooks for data labeling tasks—please let us know if you have any questions!

• Preparing an expense report
  We prompt a model to review a credit card statement, extract expenses, identify reimburseables and prepare an expense report memo: notebook

• Analyzing course evaluations
  We use a series of questions to identify, standardize and then quantify themes in a set of course evaluations: notebook

• Triaging customer service tickets
  We prompt a model to sort a collection of messages from users: notebook

• Agent-specific tasks
  We have different agent personas label specific subsets of data relevant to each persona: notebook

• Summarizing & quantifying themes in a set of transcripts
  This is similar to the course evaluations example above: notebook
  

Have a different use case for a demo notebook? Please let us know and we’ll be happy to help create it for you!

info@expectedparrot.com

Open your terminal and run the following command to install the EDSL package:

pip install edsl

To update an existing installation:

pip install --upgrade edsl

Step 2: Create an account

Sign up for a free account to access special features for conducting research at the Expected Parrot server.

Your account comes with $25 in credits for API calls to available models.

Step 3: Choose where to conduct your research

You can run surveys remotely at the Expected Parrot server or locally on your own machine.

Activate remote inference at your Settings page:

This allows you to:

• Run surveys at the Expected Parrot server.

• Use your Expected Parrot API key to access all available models.

• Automatically store and retrieve responses from the Universal Remote Cache.

Learn more about remote features.

Step 4: Manage API keys

You can use your own API keys for LLMs and/or an Expected Parrot API key that provides access to all available models.

Manage keys at your Keys page:

• Use your Expected Parrot API key to access all available models.

• Add your own keys from Anthropic, OpenAI and other service providers.

• Share keys with your team, track usage, set spending limits and easily switch between providers at any time.

Learn more about managing keys.

Step 5: Run the starter tutorial

Download and run the starter tutorial notebook to try example code. You can also explore demo notebooks for a variety of use cases at our documentation page and easily modify them for your purposes.

Have a use case that you don't see covered? Get in touch and we'll create one for you!

We’re excited to launch this blog for sharing updates about Expected Parrot and interesting developments in AI research!

We’ll also use it to share feature releases, use cases and examples for EDSL (Expected Parrot Domain-Specific Language), our open-source Python library for simulating surveys and experiments with AI agents and large language models. We love turning user questions and feature requests into demo notebooks and templates, so please let us know if you have any requests or suggestions!

Please also feel free to join our Discord and follow us on Twitter/X.