I realized fairly early in graduate school that: (1) much of empirical social science was a kind of applied software engineering and (2) that software engineering had solved many of the problems that researchers had (but generally didn’t know this). I could write a lot about this topic (Gentzkow & Shapiro's Code and Data for the Social Sciences is ‘old’ but still very good) but let me pick the most important one: version-control.

Distributed version control like git lets people safely collaborate on the same project, without worrying about data loss or (unsolvable) conflicts. Users also get the full history of changes, by whom as well as the ability to create arbitrary paths for a project that can later be used, abandoned, etc.

But to get these benefits, the object being worked on has to have a file format. And that file format should be nicely ‘diffable’ meaning that when you see the changes made over time, you should understand what was done.

However, there are two problems. The first problem: surveys have no canonical, git-friendly file format. The second: even with a good format, version control has historically demanded skills most researchers don't want to acquire. We solve the first with a new format; coding agents solve the second.

What’s done in practice now?

In research papers, surveys are often low-res screenshots of an interface or a public URL to the survey that will invariably link rot. Or they’re just described. The main reason is that there isn’t a file to be included or referenced. This is bad for several reasons:

• It’s hard for follow-on researchers to know what, exactly, you did

  • This is particularly true as surveys get more complex with skip-logic, piping, arms etc.

• It’s hard for follow-on researchers to replicate your work

  • They have to reverse engineer what you did and then reproduce it

• It causes vendor lock-in that prevents progress

  • How can I take my Google Survey to Qualtrics? Qualtrics to Google?

File-based workflows and errors

Many surveys end up just having a database representation (Google Forms, SurveyMonkey) or if they do have a file format (Qualtrics QSF), it’s not “meant” to be edited outside their web interface. It’s easier for mistakes to slip in when edits are just being made in a web interface because we don’t have access to all our normal tools for catching errors—nevermind agents.

In contrast, once you commit to a canonical file format that fully captures a survey, you can create linters and validators: a validator catching a skip rule that references a deleted question, or piping that points at a not-yet-asked question.

The problem is even more acute when data from a deployed survey is exported. Because there is no canonical file format, there is no way to provide the full survey context reliably to the analyst (now, often the agent). The core problem is that you need hacks to get non-flat data (as is common in surveys) into a flat tabular format. For example, you often see checkbox question data just exploded into N columns in a CSV with maybe one column for the “Other” free text. If the researcher—or more likely the coding agent—isn’t careful, importing this format as regular tabular data can be badly misleading.

With a canonical representation of the survey afforded by a standard file format, it could reliably decode what the answers mean. And if the research agent can see two sets of responses under different versions of a survey, it can use the git functionality to analyze if changes are seemingly making a difference.

Working with files is great, but not every researcher wants to be a software engineer!

True! And this was the reason for the proliferation of point-and-click / GUI-based tools for researchers. It’s a real and understandable trade-off: do I spend more time learning about the mathematics of linear regression or learning that regression in R is m ← lm(y ~ x, data = df) which Stata-heads find ridiculous?

HOWEVER, the rise of coding agents radically changes the calculus here and eliminates the trade-off—we can take the points made above and edit them so:

• It’s hard for follow-on researchers coding agents to know what, exactly, you did

Now we can have all the benefits of code-based research work-flows without the “minor in software engineering“ downsides.

Expected Parrot Solution: Decompose a survey into a collection of smaller files and put git right in the format

Most written programs are text files and line-level “diffs” are fine to illustrate meaning. Data structures like surveys are more complex. You want the “diffs” to be semantic and narrowly scoped is possible. But this is hard if the “Survey” is just some big blob of nested JSON. With this format, it would be harder to understand what, exactly, changed from commit to commit.

Our approach at Expected Parrot / EDSL is to store a survey as a collection of small files. We have one per question in a survey (under questions/):

example-survey.ep                   one portable ZIP-compatible file
└── unpacked contents               a complete Git working tree
    ├── .git/                       commits, trees, blobs, refs, and configuration
    ├── manifest.json               package type, format version, and question order
    ├── questions/
    │   ├── 000001.json             one question per file
    │   ├── 000002.json
    │   └── 000003.json
    └── metadata/
        ├── memory_plan.json
        ├── question_groups.json
        └── rule_collection.json

This way, if a survey has 500 questions and one changes, Git does not need to treat one giant serialized survey document as the unit of review. The diff can focus on the affected question and any metadata that actually changed.

Why an ‘ep’ file format that’s a zipped directory?

Note also that by putting .git right in the specification, the history of the survey can travel with it. There is a current survey, but it also contains the whole history of that survey.

But note that we zip it—doesn’t that defeat the purpose? No. A survey likely lives inside a larger research project. Nesting git repos is ill-advised, so instead we zip the repo and use the .ep suffix. Yes, the “outer” project repo sees an opaque file, but the survey's own history is complete and portable inside it, and you can unzip to inspect. But more importantly, your agent knows EDSL and thus “knows” how to use it (e.g., when it makes changes to a survey you direct, it creates internal versions), giving you all the benefits of git without having to “shell out” to use git. And to be clear, you can unpack the file if you do want to poke around. You can even push that repo to GitHub / GitLab etc.

Decomposition makes for efficient, legible changes

Some files under preserve behavior that is not contained in an individual question: The manifest.json identifies the format and reconstructs question order:

{
  “edsl_class_name”: “Survey”,
  “edsl_version”: “1.0.8.dev1”,
  “format”: “edsl.survey.git_package”,
  “format_version”: 1,
  “n_questions”: 3,
  “object_type”: “Survey”,
  “question_order”: [
    “000001”,
    “000002”,
    “000003”
  ]
}

What’s good about this is changing, say, the order of questions in a survey, is also not a massive diff that changes every line: it changes one line in manifest.json. This is more efficient (and will matter more when we allow multi-party editing of a survey in real time). Compare this to a monolithic JSON file, where reordering questions rewrites the entire array and the diff is useless.

The repo also has other files in the package that capture other aspects of a survey, such as skip-logic, flows, question groups and so on.

Showing a diff: A changed question text

Each question is stored separately. For example, questions/000001.json contains:

{
  "question_name": "q0",
  "question_options": [
    "yes",
    "no"
  ],
  "question_text": "Do you like school?",
  "question_type": "multiple_choice"
}

Now suppose we make an edit (in Python, but your coding agent could be doing this work):

survey = Survey.git.load("example-survey.ep")
survey.questions[0].question_text = "Do you enjoy school?"

second_save = survey.git.save(
    message="Clarify the first question",
)
second_save

If we look at the log for the object, we can see the new commit, just like a regular git repository:

survey.git.log(...)

[
    {
        "commit": "d1f44ff624288e48e8835597d247c039ceffeb48",
        "message": "Clarify the first question",
        "timestamp": "2026-07-29 07:47:56 -0400",
    },
    {
        "commit": "65801c7e03eb93c0f31c8bee6b2a29cc9152be14",
        "message": "Create example survey",
        "timestamp": "2026-07-29 07:47:34 -0400",
    },
]

EDSL exposes the git semantics for looking at a diff:

print(survey.git.diff("HEAD~1", "HEAD"))

And we can see the familiar git differences like so:

diff --git a/questions/000001.json b/questions/000001.json
index 353b419..33b47da 100644
--- a/questions/000001.json
+++ b/questions/000001.json
@@
-  "question_text": "Do you like school?",
+  "question_text": "Do you enjoy school?",

Explore alternatives with branches

Branches let collaborators develop alternative instruments without overwriting the main line:

survey.git.branch(”short-form”)
survey.git.switch(”short-form”)

# Make edits to the in-memory Survey here.
survey.git.save(message=”Create short-form variant”)

Return to the main version:

survey.git.switch(”main”)

switch() refreshes the in-memory Survey from the selected branch. EDSL requires a clean package before switching, pulling, or checking out another version, which protects uncommitted changes.​

Merge a survey branch

Suppose the short-form branch removes the follow-up question q1:

survey.git.switch(”short-form”)
survey.delete_question(”q1”)
survey.git.save(message=”Remove follow-up from short form”)

At this point, main still has all three questions and short-form has two. Switch back before merging:

survey.git.switch(”main”)
survey.question_names

[”q0”, “q1”, “q2”]

Merge the branch through the same Git-backed accessor:

merge_info = survey.git.merge(
    “short-form”,
    message=”Merge short-form survey”,
    no_ff=True,
)
merge_info

{
    “status”: “ok”,
    “path”: “example-survey.ep”,
    “commit”: “<merge-commit>”,
    “branch”: “main”,
    “merged_ref”: “short-form”,
    “fast_forward”: False,
    “changed”: [
        “manifest.json”,
        “metadata/memory_plan.json”,
        “metadata/rule_collection.json”,
        “questions/000002.json”,
    ],
    “conflicts”: [],
    “aborted”: False,
    “message”: “Merge short-form survey”,
}

no_ff=True asks Git to create a merge commit even though this simple example could be fast-forwarded. That makes the point where the alternative survey entered main visible in history.

Ok, how do I use this in my work?

You don’t really have to “do” anything - it’s just what is happening under the hood when you use EDSL and our research agent for your survey work. However, a coding agent working with EDSL understands these capabilities and so you get the benefit of a git-based format without learning a great deal about our plumbing.

Want to try it out?

Use Expected Parrot

tl; dr - A recent, great paper discusses the problem of randomness in LLM responses and the problems in creates for research. EDSL / Expected Parrot deals with many of these issues.

This is worth reading:

Many of the problems the paper identifies motivated certain design features in our Python package, EDSL. Rather than write out the connection myself (and insert my bias), I put Codex on the task, launching it from the current edsl repo directory:

Below are the responses, with some editorial comments from me (JJH), but the short version is Codex nails it:

<🤖> The paper’s central argument is that LLM-generated measurements should be treated as draws from a distribution, not as fixed data. EDSL is unusually well aligned with that view because it represents the entire data-generation experiment—prompts, cases, models, repetitions, and outputs—as structured objects.

EDSL makes repeated measurement a first-class operation

The paper recommends measuring output variability instead of assuming that one call is representative. It specifically suggests repeated calls, potentially on a documented subsample when cost is a concern. The paper’s experiment repeats each classification 200 times, and its reporting standard asks for the number of draws, agreement measures, and sensitivity of substantive estimates. In EDSL, repetitions are part of the job specification (JJH: and can be run from the command line or executed with a Python script):

ep run jobs.ep --n 200 --fresh --output results.ep

JJH: A “job” in EDSL jargon is a survey x scenario x agent list x model list. This can be serialized to an “ep” format, which essentially a git repo, zipped up (giving us version control on the object).

The resulting data contain an iteration.iteration field, so each draw remains separately identifiable. Researchers can then export the repeated responses and calculate agreement rates, empirical response distributions, or a distribution of downstream regression estimates.

The --fresh detail matters: a variability study should explicitly avoid substituting cached answers for new draws. EDSL records whether a response came from cache and its cache key, making this auditable.

It preserves the prompts as executable, machine-readable artifacts

The paper says “prompts are code” and argues that screenshots are inadequate because they cannot be executed, searched, or compared. It calls for verbatim user prompts, system messages, and examples in the replication package. EDSL saves the actual rendered prompts with every result, including separate fields such as:

- prompt.q0_user_prompt - prompt.q0_system_prompt

This is better than saving only a prompt template. If a question contains scenario values, agent traits, prior answers, or survey instructions, the result contains the prompt that was actually sent for that observation. The underlying Survey, AgentList, and ScenarioList preserve the reusable experimental design as well.

That gives a verifier both:

• the source specification used to regenerate prompts; and

• the observation-level rendered prompts used in the original run.

JJH: Another way of putting it is that EDSL preserves both what was actually sent to the model as a prompt, as well the template it was constructed from, as well as the data used to populate that template.

A saved Jobs object captures the complete experimental design

The paper asks replication packages to provide a route for rerunning the model-generation step. An EDSL Jobs object bundles:

• the survey and its question wording

• survey instructions and flow;

• agents and their instructions or traits;

• scenarios containing the input data;

• models and their generation parameters.

A durable .ep package can therefore serve as the machine-readable specification of the LLM measurement procedure:

  ep validate --file jobs.ep
  ep inspect jobs.ep
  ep run jobs.ep --fresh --output regenerated-results.ep

This reduces a common source of replication failure: code, prompt files, input data, and model settings drifting apart.

Model configurations distinguish provider, model, and parameters

The paper’s minimum reporting standard includes the provider, exact model identifier, access route, and generation settings—especially temperature. EDSL model specifications serialize the inference service, model name, EDSL version, and model parameters. For example:

{
    “model”: “gpt-4o”,
    “inference_service”: “openai”,
    “parameters”: {
      “temperature”: 0,
      “top_p”: 1,
      “max_tokens”: 1000
    }
}

Researchers can construct heterogeneous model lists without hiding those choices in application code:

ep models create \
 --model-spec ‘{”model”:”model-a”,”service”:”provider-a”,”parameters”:{”temperature”:0}}’ \
 --model-spec ‘{”model”:”model-b”,”service”:”provider-b”,”parameters”:{”temperature”:0}}’ \
 --output models.ep

This supports both transparent reporting and robustness checks across models or providers. It also helps researchers follow the paper’s recommendation to use temperature zero when sampling is unnecessary—without implying that the result is deterministic. As the paper emphasizes, temperature zero does not eliminate silent model updates, batching effects, floating-point variation, or expert routing.

Results preserve raw output alongside processed answers

The paper calls the deposit of every raw model output “the most important item” in a replication package. That permits exact reproduction of downstream analyses even if the original model changes, becomes unavailable, or becomes prohibitively expensive.

EDSL Results distinguish among:

• the parsed answer;

• the generated text;

• the provider’s raw response;

• validation status and comments;

• token usage and cost.

For example, a result can contain both answer.q0 and raw_model_response.q0_raw_model_response. This separation is important: a researcher can archive the original provider response, inspect how EDSL parsed it, and rerun downstream cleaning or analysis without making another model call. This directly supports the paper’s two proposed verification paths:

1. Reproduction from archived outputs: take the saved EDSL Results as given and rerun parsing/statistical analysis.

2. Fresh replication: rerun the saved Jobs specification and compare the new draws with the archived distribution.

JJH: All model outputs are stored in our Universal Remote Cache and so if someone else tries to replicate your paper, they can do so at no cost: when they re-run, they’ll hit the cache, which doesn’t cost anything.

EDSL exposes caching rather than making it invisible

Caching is valuable for ordinary reproducibility because it lets a researcher recover previously generated measurements without paying for another call. But it can invalidate a variability experiment if cached responses are mistaken for independent model draws.

EDSL addresses both needs:

• fresh requests genuinely new inference;

• Results record cache_used and cache_keys;

• saved outputs allow the original response to be reused deliberately.

This makes the distinction between “reproduce using the archived measurement” and “draw a new measurement from the model” explicit—the same distinction made by the paper.

It supports systematic sensitivity analysis

The paper shows that LLM variability can propagate into downstream regression coefficients and even change whether a result crosses a significance threshold. EDSL makes relevant experimental dimensions composable:

scenarios × agents × models × iterations × questions

Researchers can therefore vary:

• temperature or other generation parameters;

• model and provider;

• prompt formulation;

• agent instructions;

• input cases;

• repeated draws.

Because these dimensions are stored as structured result columns, outputs can be grouped and compared without reconstructing experimental conditions from filenames or logs. This is a strong foundation for agreement statistics and specification-curve-style sensitivity analysis.

It makes cost reporting and affordable pilot studies easier

The paper treats token counts, monetary cost, and runtime as required replication-package information. It also recommends using repeated calls on a subsample when full repetition is too expensive. EDSL provides:

ep jobs cost jobs.ep --iterations 20
ep results cost results.ep
ep costs log --output costs.jsonl --actual-from results.ep  

Individual Results include input tokens, output tokens, price information, and calculated cost. Researchers can estimate a repeated-measurement design before running it, conduct a pilot on a smaller ScenarioList, and retain an actual-cost ledger afterward.

Structured inputs improve provenance

For annotation and extraction research, EDSL scenarios provide a stable mapping between an input observation and its model calls. Rather than interpolating documents into ad hoc strings and saving only the response, a researcher can store document identifiers and content in a ScenarioList. Results then retain the scenario, rendered prompt, model configuration, iteration, and response together.

That makes it much easier to answer:

• Which input generated this answer?

• Which exact prompt was sent?

• Which model configuration handled it?

• Was this the first or twentieth draw?

• Was the answer retrieved from cache?

• What raw response was parsed?

Important limitations

EDSL helps implement the paper’s standard, but it cannot solve provider-side nondeterminism. In particular:

• EDSL cannot preserve proprietary model weights or prevent silent updates.

• A provider’s public model name may not be an immutable checkpoint.

• Hardware, batching, routing, safety configuration, and provider fingerprints are only available if the provider exposes them.

• The Results fields I inspected do not establish that query date/time, exact provider-returned model version, fingerprint, or total wall-clock runtime are universally captured.

  • JJH: Good point! We’ll add all this!

• EDSL does not by itself containerize a local inference stack or archive weights and their cryptographic hashes.

• It facilitates variability analysis, but researchers still need to calculate and report agreement measures and the sensitivity of their substantive estimates.

  • JJH: We can/should build a package for doing this to sit on top EDSL.

• So the most accurate claim is: EDSL does not make proprietary LLM inference reproducible. It makes LLM-based research substantially more transparent, archivable, repeatable, and auditable—and gives researchers the machinery to measure irreproducibility instead of concealing it.

That is very close to the paper’s conception of best practice. The strongest opportunity for EDSL would be a dedicated “reproducibility report” export that automatically summarizes model identifiers and parameters, rendered prompts, query dates, cache use, token counts, costs, repetitions, agreement statistics, and missing provider metadata against the paper’s Table 2 checklist.

</🤖>

Want to use EDSL in your own workflows?

Try Expected Parrot!

tl;dr - I had an idea for a <robot> tag that authors could use to identify AI-written passages in blog posts and thought it might be a useful new convention that increased readers’ trust. My simulated agents agreed, but my real human study threw some cold water on it.

The other day, I had an idea for a way that authors could responsibly use AI in their writing. If an author clearly fenced off AI-written text with “robot” tags, perhaps that would solve some of the problems with so-called “AI slop.” Readers would appreciate the candor and increase their trust in the whole thing:

The idea is you’d see passages like this:

<🤖> Sure, I’ll prove the Riemann conjecture for you…</🤖>

I had actually already used this approach in several blog posts. There seems to be a real gap between how much people dislike or claim to dislike “AI slop” and their willingness to read AI-generated text when it’s useful and presented appropriately.

Could the robot tag idea be a paper?!

As the likes and re-tweets rolled in, I started thinking, “Hmm, maybe I could write a paper about this.” Of course, it’s not a “big idea” and the potential upside, research-wise, of introducing a new kind of blog post tag isn’t that high. It’s not the kind of thing I’d sink a ton of time into unless I had some really, really easy way to do the studies…

Let’s use the Expected Parrot Research Agent

But I do have an easy way to do studies! I started up the Expected Parrot Research Agent and described what I wanted to investigate, referencing the original tweet. My full Research Agent transcript is here.

In the transcript, you can see we actually do several different things: explore the idea, play around with framing, develop questions and stimuli, run pilots, revise the study, etc. There’s a part where I tweak the CSS of the “robot” tag in the study to be sent to human participants. One key thing about the Expected Parrot platform is that it’s designed from the ground up for taking a survey given to LLMs and then administering it to real humans.

Collaborator, not a paper factory

If you read the transcript, you’ll see there is a lot of back-and-forth where I propose ideas, critique suggestions, commission pilots, tweak language, etc.:

This is the intended workflow. The Research Agent is not a push button that produces a finished paper. It’s a research collaborator that makes it much faster and cheaper to explore an idea, test assumptions, and improve a study before spending real money on participants.

Promising results but in silico and an underpowered human pilot!

I ran a little pilot with simulated subjects that suggested the idea might be promising. I then ran an underpowered study on Prolific (I’m not made of money). Effects went the “right” way. I could already see the academic paper: maybe not a world-changing piece of scholarship but a nice solid base hit. A contribution squarely in the Information Systems tradition…

Ah crap…

On the basis of my promising pilots, I launched a larger study on Prolific. And when I ran the real study, nothing happened. At all.

Has my robot-tag idea been disproved?

No. I still think it’s a good idea. But maybe not the slam dunk I initially imagined. Maybe I’m now a bit sadder and wiser, but social science studies rarely “prove” anything, one way or another. I could think of ten different reasons why this study doesn’t really speak to the power of the robot tag. But the study is still evidence, and every study provides information that should update our priors on our own imperfect models of the world, even if it doesn’t settle the question. We’re all in this messy business of taking in new data and computing the cross-entropy with respect to the model we already had in our heads. Treating every result as either total confirmation or complete refutation is childish. A more useful question is: How much should this result change what I believe?

That is especially true outside academia, where the goal is often to make a decision rather than publish a paper. Even a well-designed study rarely captures every relevant feature of the real-world problem. But it can still make the decision better.

In this case, the evidence moved me from “This obviously works!” to “This (still) seems plausible but could use more work.” That is definitely not as exciting as proving myself right, but it is probably more useful.

Test your own idea

One reason we built Expected Parrot was to make this kind of small, speculative research project easy to pursue. You can explore an idea, test it with AI, and then run the same study with real people without turning it into a major undertaking.

So, if there is a question you have been curious about, please try it on Expected Parrot and request access to our Research Agent. If you need any help, please get in touch!

Yesterday Amir presented our paper, “Marginal Ideas,” at the NBER Summer Institute Entrepreneurship session. Here’s a video of Amir giving the talk and Scott’s discussion.

The paper conceptualizes entrepreneurship as a kind of congestion game where anyone can pursue any idea they want. Entrepreneurs have a guess as to how good an idea is, but don’t know for sure unless someone pursues it. This causes entrepreneurs to spread themselves out over ideas, with good ideas attracting more entrepreneurs than seemingly bad ideas. In equilibrium, it’s the “marginal idea” that matters: the worst idea that still gets funding.

You can read the paper if applied microeconomic theory is your thing, but as this is the Expected Parrot blog and not the “cool tricks with supply and demand curves” blog, let us explain the empirical portion. The paper’s big problem is: How do you identify when startups are pursuing the same idea?

Birds of a feather (bid on the same keywords) together

If two startups are bidding on the same keywords to boost their websites in Google searches, they are likely competing for the same customers. Google’s ad auction data (if you can get it, more on that below) is especially nice because it shows what economists call a “revealed preference.” It means exactly what it sounds like. Imagine you want to infer whether John likes strawberry or goat cheese ice cream. He claims he would never eat goat cheese ice cream, but you can also observe what he actually pays for when he visits Toscanini’s! Same idea. Startups pay real money, per click, to intercept a specific customer intent, so two startups bidding on “dog walker near me” are rivals in the only sense that matters economically, whatever their websites may say.

The idea in one line is: Get the keywords, then cluster startups based on the similarity between these keywords.

Problem 1: Most startups don’t actually buy keywords

So why not just look up what startups actually bid on? We tried. Two problems. Most startups never run paid search at all, so the ones that do are badly selected—perhaps they have more money, or think they have a better idea. That’s a big problem because we don’t want to learn about a select group of startups who are different for reasons we don’t understand. We want to know about the population of startups. But, even if all startups did have real marketing keyword bids, we need to buy this data from vendors that sell historical bidding information. Well, even after negotiating the “research use” quote with one of the largest vendors of these data, the price tag made clear it was probably meant for hedge funds, not academic research.

Problem 2: No 10-Ks for Startups

The standard academic move doesn’t work here either. The best existing measures in Industrial Organization—the canonical field concerned with competitive forces of the economy—are built from the legally mandated business descriptions of public firms, obtained from their Form 10-Ks. These are fairly long, standardized, and legally required to be accurate. (Now you know how to find out what Palantir does.) For startups, though, there is no legally mandated anything for product descriptions. All we have is a blurb from Crunchbase, possibly written by the founders; sometimes marketing oriented, other times surprisingly puzzling.

Problem 3: Semantic/topic similarity isn’t good enough for our purposes

Here’s one description from our data: “Application: Uber for dog walking. Hardware: Fitbit for Dogs.” You, a human, read this and think: okay, dog care with a health tracker! Every NLP method we tried though reads these words and matches this either with ridesharing platforms or wearable devices. The same method groups DoorDash and Instacart together because of the obvious delivery and food keywords in their descriptions, but one focuses on restaurant delivery and the other on grocery delivery. They are not direct competitors.

Enter the Stochastic Parrots (LLMs)

To get the keywords for all startups, we generate them using AI. We give an LLM agent the role of a Google AdWords specialist and hand it nothing but the vague blurb, then ask it to draft the ten non-branded search keywords this firm would bid on to reach customers who are already searching for a competing product. Figure D1 below from our paper shows the exact prompt.

Note that not all keywords are created equal! We can safely assume that DoorDash is far more willing to pay for “restaurant delivery” than for “grocery delivery,” and for Instacart it’s exactly the reverse. We also ask the same AdWords specialist to decide what fraction of their marketing budget they’d allocate to each keyword, mimicking what a human bidder does. The key point here is that the specialist isn’t asked what the company is, it is asked what the company would pay for.

We run this as an Expected Parrot survey over every startup in our data. In short, we chose E[P] over direct API access because it makes experimenting, documenting, and large-scale piping of LLM surveys much easier. Here are a few examples of how it helped us.

We ran the same prompt across models from different providers and across different agent personas, in parallel, so figuring out what configuration produces best results was a couple of hours on a single script rather than an engineering project.

Another advantage is that the answers come back in the shape you asked for. Ask for a list of exactly ten keywords (note the max_list_items and min_list_items in Figure D1 above) and you get a list of exactly ten keywords. You can choose from several survey types, including Likert scale and dictionaries. Anyone who has regex-parsed “Sure! Here are your keywords:” out of thousands of raw API responses knows this is not a trivial problem.

But what about…

Now, we anticipate that you are screaming “Garbage in-garbage out!” How do we know the keywords are good? The short answer is by “standing upon the shoulders of giants.” Recent work by economists on the frontier of AI and social sciences shows that LLMs excel at filling latent details from sparse inputs and predicting human behavior obtained from gold-standard randomized experiments.

Of course, we also do our own due diligence. To assess the risk of “garbage in,” we manually read predicted keywords from multiple randomly selected samples. To assess the risk of “garbage out,” we hand-label a subset of data (think of this as our test set) and assess the final clustering outcomes against it. As we’ll see, the results are quite good!

From keyword portfolios to markets

We are one final step away from finding out who’s competing with whom. We just need to take these keyword portfolios and find pockets in which there’s “a lot of” overlap. The remaining challenges are fairly straightforward technical tasks. If you are familiar with these terms, we use a mutual k-NN graphs with Leiden clustering and find the model parameters using a hand-labeled dataset.

If you are not familiar with these terms, here it is in English. To turn pairwise overlap into markets, each firm nominates its closest competitors, and then we keep a link only when the nomination is mutual. By now, we have a massive hairball with almost all startups connected to each other, either directly or indirectly, and some strongly with lots of keyword overlap, and some weakly. To chop this graph up into small communities of competing startups, we use an algorithm that ensures each cluster’s members are internally connected and clusters pay a price for their size (otherwise, you have no way of knowing where to stop).

It works. Against our hand-labeled data, about 90% of the niche-mates assigned by the model are genuine market-mates (precision), and the niches capture about 80% of their market-mates (recall).

And the dog thing startup? Turns out there is more than one “Uber for dog walking” in our data, which, yes, is rather the point of the paper. There are also quite a few pet-wearable startups, and the startup with the puzzling description, was clustered with them (with Uber and Fitbit nowhere near to be found).

What did we find??

Read & cite the paper.

Despite the true value of “learning from failure," all else equal, it’s better to just succeed right away. Of course, “do things likely to succeed” is the buy-low-sell-high of planning advice. What is to be done to actually make success more likely or choose better paths? One approach is to just think real hard about what might go wrong and then either scrap the idea or plan various mitigations. This is a good idea, but sometimes it’s hard to be truly critical of an idea if you’re excited about it, and you often lack the full context to understand why failure might arise.

Getting true critics is challenging

What you really need is other people to help crap on your idea, especially if those people bring a different, relevant perspective. But in any actual team setting, this creates some other problems because “person who craps on other ideas” is almost the opposite of being a team player. As a case in point, I once worked on a team where one of the engineers had a reputation for finding reasons why something was a bad idea or impossible. A teammate said one of the most damning putdowns I’ve ever heard: “Joel is the world’s expert in what cannot be done.” Ideally, you can present an idea to the team that is already thought out, with most objections planned for, that getting consensus isn’t a slog (or you’ve nipped the bad idea in the bud).

The “pre-mortem” management technique

As a business school professor, I have an encyclopedic knowledge of various management techniques and by that I mean “I looked up on Wikipedia” the idea of a pre-mortem. The idea is quite simple: you get a group of people together—relevant stakeholders—and assume the initiative has already failed. The only thing left is to figure out why. One of things that is nice about this approach is that people have permission to play the role of critic. After you collect the reasons for failure, you can do some fancier stuff to trace out the causal pathways. But people are busy and meetings are expensive.

Enter simulated AI naysayers

This “have people play the role of critics” doesn’t necessarily require real people. In fact, you can have AI play those roles, get their (very detailed) feedback and then do the same kind of analysis you might do with real people. The big advantages, however, are:

• The time of the AI fake people isn’t worth very much—you can get their views for pennies.

• You can get their views with guaranteed, assured privacy.

• You can try lots of iterations without the AI people wanting to strangle you for boring them or wasting their time.

Why Expected Parrot for this?

You could try this in a normal chat interface, but there are limitations. Let’s ask Claude for a full explanation of why you should use us instead:

<🤖> Why use Expected Parrot rather than one-shotting with Claude

Independent agents vs. one voice. In a single chat, one model generates all “personas” in a single pass, so they contaminate each other — the critiques converge and echo. Expected Parrot issues a separate API call per persona, so each reasons independently. You get genuine variance instead of one model’s averaged impression of what a crowd would say.

Multi-model. A chat is locked to one model. A survey framework lets you run the same questions across different models, which surfaces disagreement and reduces any single model’s idiosyncratic blind spots — closer to real panel diversity.

Structure and reproducibility. It’s a framework built for this: defined personas, consistent question batteries, structured (parseable) outputs, and the ability to re-run with tweaks. One-shot chat output is prose you’d have to manually wrangle; here you get data you can aggregate, compare, and version.

Scale and iteration. You can run dozens or hundreds of personas and many iterations cheaply and systematically, rather than repeatedly re-prompting and losing track of what changed.

Reusable digital twins. As the post notes, personas can be grounded in real survey or observational data to build persistent “digital twins” you reuse across studies — something a fresh chat can’t retain.

Tooling and skills. The agent chains web search (the Firecrawl plugin) and packaged skills like the pre-mortem workflow into the run automatically, so the critique is grounded in current external info rather than the model’s static knowledge.

The short version: a chat gives you one model’s impression of what critics might say; Expected Parrot gives you a structured, multi-agent, reproducible simulation of critics — which is what makes the output feel like data rather than a monologue.

</🤖>

Using the Research Agent to kill our sailing kayak dream

Robin and I were thinking about getting a sailing pedal kayak. They look really fun: you pull right up on the beach (no keel or centerboard); you can paddle, and your legs are stronger than your arms; and if there is a breeze, you can take advantage of it. But, they are not cheap! We were ready to buy, but I thought, “eh - maybe I should see what our Research Agent thinks.”

From expectedparrot.com we launched a session and asked for a pre-mortem on this potential purchase. Note that I included a bunch of context that will help the Research Agent do something smarter and less generic. I explained what other boats I already have, our personal preferences, where we live and so on.

The Research Agent starts immediately doing some web search using a plugin we have with Firecrawl (this makes scraping more efficient than doing direct web searches):

Another thing you can see in the tool calls that is leans on a skill / package we’ve created for doing these kinds of pre-mortem exercises.

It sets up personas for our AI critics:

Where are these personas coming from??

The research agent just makes up personas that are relevant to the decision problem at hand. Though, critically, you could create your own digital twins from past surveys (either your own or others), observational data you have, etc.

The report

The report is here. It’s quite interesting and raises some considerations, some of which I agree with, some of which are oversold. For example, it says the tides are a big issue—they aren’t. It drew a diagram showing how everything failed:

The root causes are worth thinking about. 1 isn’t a big deal—it overstates the tides. 2 actually is worth thinking about because it doesn’t have a dolly. But 3 stopped me dead in my tracks!

You need to rinse these things with fresh water after every use! Or as Derek puts it:

We don’t have freshwater where we could store them! This is a total deal-breaker. Purchase canceled!

We, as a company, do not aspire to be the #1 tool for hybrid watercraft purchase decisions. The point of this example is to show-off the technique and the kind insight this can generate. But what might be commercial applications? Let’s go back to Claude:

<🤖> The most valuable commercial use isn’t personas reacting to things in general—it’s aiming that simulated-critic machinery specifically at initiatives a company has already decided to pursue, precisely when enthusiasm and sunk commitment make honest internal criticism scarce.

• A retailer about to sign a lease for a new location can run a pre-mortem in which simulated store managers, local competitors, and target shoppers assume the store failed within a year and explain why—surfacing the freshwater-rinse-level dealbreaker (no parking, a competing anchor tenant leaving the mall, foot traffic that evaporates after 6pm) before the lease is signed rather than after.

• A SaaS team convinced a new feature will drive retention can assume it launched and nobody adopted it, then let simulated power users, skeptical admins, and budget-conscious buyers reconstruct the failure path: it collided with an existing workflow, IT wouldn’t approve it, the value wasn’t obvious in the first session.

• A pharma or hardware company can pre-mortem a launch, a marketing team a campaign that “already flopped,” a PE firm an acquisition that “already went sideways,” an HR team a return-to-office policy that “already triggered attrition.”

The through-line is that each of these is a decision the organization is emotionally and financially committed to, where a real pre-mortem is exactly the right tool but expensive to convene and awkward to run honestly—and where simulated critics let you trace the causal pathways to failure cheaply, privately, and as many times as it takes to either kill the idea or harden it.

</🤖>

If this sounds like you, sign-up for an account and try it out. It comes with $25 in free credits!

tl;dr - Using LLMs to explore historical documents is a fantastic application and Expected Parrot makes it even better/easier.

Around the 4th of July my family and I saw a performance of the musical 1776, in which John and Abigail Adams play an important role. The Massachusetts Historical Society has digitized much of their correspondence. While it is convenient they are already digitized, vision-capable LLMs are remarkably good at reading old handwriting:

Her letter reports some bad behavior by the British (namely abusing a nice mahogany table and curtains/cushions):

This reminded me of one of my favorite Onion bits: “This is a proud day for quarters-owners everywhere,” said the organization’s president, Charles Davison, in his keynote address. “Year after year, we have sent a loud and clear message to the federal government and to anyone else who would attack our unassailable rights: Hands off our cottages, livery stables, and haylofts.”

This little excerpt gave me an idea: I should go through a bunch of her letters and compile her reports on British activity.

There’s already a growing recognition among economic historians that LLMs can enable the use of data that otherwise would have been too expensive to use. That being said, using LLMs in this way forces researchers to play software engineer. And while coding agents help, there’s a better way: the Expected Parrot research agent and write the code to do this kind of work based on natural language. You might be thinking: wait - isn’t EDSL / Expected Parrot something for surveys? Yes, they are, but it turns out that the survey paradigm—with structured question types, piping, skip-logic, multi-modal/multi-model support, etc.—is also phenomenal for setting up and verifying complex labeling flows with LLMs.

Expected Parrot Workflow

To start, I had Codex scrape the Historical Society website for letters during the Continental Congress years. Then I uploaded the letters to the Research Agent as a zip file and explained what I wanted done:

After doing some exploration and digging, the Research Agent asks me some questions to clarify my goals and the scope:

I also asked for multiple models (Expected Parrot supports a huge number out of the box):

comes up with a plan for me to review:

A key part of the plan is the skip logic: a screening question to first see if the letter mentions British behavior and then, if it does, have it label from a list of activities. It’s smart enough to use a checkbox question here since one letter could mention several kinds of activities.

It also helpfully added a question to summarize the letter contents with respect to British activities in a 3rd question:

Note also the mustache scenario.letter_text in the prompt. This is expanded to when sent to the model with the actual scenario (in this case all the letters). This keeps prompts legible and the labeling instructions/context separate from the raw data. The actual ScenarioList is constructed like this. A key point for reproducibility/legibility is that the coding agent is writing Python files to implement this labeling flow:

Because EDSL has a declarative syntax, the actual labeling job sent to Expected Parrot’s services is concise:

Upshot: A written report on what we learned

The Research Agent executes the labeling flow and analyzes the data:

It doesn’t hide areas of model disagreement but instead surfaces them in helpful comparison visualizations:

If you wanted to, you could then send some sample of these letters to be labeled on Prolific. Another upshot of using coding agent for this task is that you can have it create custom HTML pages on the basis the data and share them with others. To me, it was natural to want an interactive timeline, which you can play with here:

As our research agent is based on a general coding agent, you can easily have it make bespoke HTML pages like this and then share them.

Commercial Applications!

As we, Expected Parrot, do have investors who would probably not be excited about the “#1 Tool for Historical LLM based-workflows” let me ruin my perfect Pangram rating for everything above by asking Claude Fable what, based on the above, would be the commercial applications of this tech:

<🤖> While EDSL is an open source package, Expected Parrot is a company—and the same workflow that made quick work of Abigail Adams's letters is exactly what production teams need. Swap the correspondence for insurance claims, contracts, support tickets, clinical notes, or survey open-ends, and the pattern holds: a screening question, some skip logic, a checkbox list of categories, and a summary—now running across thousands of documents and several models at once. The features that felt like nice-to-haves on a hobby project turn out to be the whole ballgame in commercial settings. Structured question types and declarative syntax keep the labeling logic legible and auditable. Reproducible Python files mean your pipeline isn't trapped in a chat window. And the multi-model disagreement visualizations do real work: instead of asking a stakeholder to trust a black box, you can measure where the models agree, route the contested cases to human reviewers on Prolific or internal teams, and hand over something defensible. That last part matters most. In regulated or high-stakes work—underwriting, compliance review, adverse-event abstraction—"the LLM said so" isn't an answer. A transparent flow that shows its screening logic, surfaces its uncertainty, and keeps a human in the loop for the hard cases is. Turns out the tool built for surveys was built for this all along. </🤖>

Want to try this out? Sign up at expectedparrot.com. Accounts come with $25 in free credits and you can earn more by referring others. Also, this just scratches the surface of what’s possible.

tl-dr; If you run consequential surveys, you should be testing them with simulated respondents ahead of time to identify problems with the survey at both a technical level and a research question/conceptual level. Expected Parrot makes it easy to do to this.

I was once involved in fielding a survey to software developers about their AI tool usage. The survey asked if they user had used “GitHub Copilot”; the survey methodologist we were working with had never heard of this and edited it to just “GitHub” assuming we had a typo. We missed his edit and as a result, nearly 100% of our developers reported “Yes.” Oof.

I think nearly researcher who has run surveys knows this feeling: the data comes back, and a question that seemed clear turns out to have meant three different things to three different people; a skip pattern didn’t work right; a scale confused everyone; nearly everyone selected “other” which is not what you planned. By then it’s too late. You’ve wasted money & time. Even if the survey is fine, there is the similar feeling of “ah, we should have asked X” once you actually start doing the analysis.

Cognitive testing / piloting are great but…

The known defense against this kind of failure is piloting and “cognitive testing”: run the survey past some real people first, watch where they get confused, and fix it. The problem is that good cognitive testing is slow and expensive. If done with a convenience sample, it also requires someone playing a role—actual target population of a survey—and it’s not likely we really understand this population with respect to the question (if we did, we probably wouldn’t be asking).

AI gives us two nice options to this slow, expensive and unrepresentative problems. We can now use AI to proofread a survey. Frontier models have some good practices “on distribution” i.e., they know a survey shouldn’t have a double-barrelled question. They know that people will tend to choose middle options and so on.

Suppose you are surveying customers and you draft this:

Q7. How satisfied are you with the support you’ve received?

○ Very satisfied

○ Satisfied

○ Neutral

○ Dissatisfied

○ Very dissatisfied

You can give this to a frontier AI. Claude, for example, nails the issues:

But here’s the rub: If I am editing a survey on Qualtrics or Survey Monkey, how, exactly, do I get Claude to review? Maybe Qualtrics has some homebrew AI tool but it’s likely behind a paywall you’re on the wrong side of. And suppose it makes some nice fixes, how can I deploy my survey?

Not being able to deploy it forecloses an exciting option in design. We can have AI simulated personas take our survey. To the extent those respondents are realistic, we can even analyze the data they return to pilot / stress test our whole analysis flow. But if I want to have simulated personas try out my survey, how do I do that? Do I give “them” a link to my survey? Paste the survey text into chat and the persona? How do I get the data out? I can ask Claude to simulate, but it will just “one shot” a plausible-seeming distribution of responses not actually have a model pretend to be a given persona.

If I want to do more sophisticated analysis on the responses, am I manually downloading a CSV giving it to Claude Code? If I learn something, I need to go back to Qualtrics or Survey Monkey. It sounds like a flow-breaking hassle.

And more importantly, it breaks the survey design process into distinct design & test phases where instead it should be happening at the same time. Imagine programming computers but you can only test the code you’ve written it and it’s expensive to do so.

The Expected Parrot Approach

We’ll use the Expected Parrot Research Agent as a design studio for our survey. It’s a conversational agent so you can just paste in what you want to do:

The research agent asks a bunch of clarifying questions to understand my intent and goals:

The research agent does not just keep all this in a chat context—it writes Python code in our open source package EDSL to create, in code, the survey being used, the personas being constructed, the data-generated and so on.

The personas here are just reasonable guesses but if you have past survey data, something a lot smarter can be done instead.

The research agent actually runs these personas through the survey using our API. Critically, EDSL has a standard for all the usual question types so the AI simulation very closely matches what would be send to actual humans e.g., free text, multiple choice, text-box and so on.

After this simulation, it finds the same issues as Claude (this is a pretty simple one) but it’s done by those AI personas actually going through the survey. This becomes critical with more elaborate personas, complex piping and skip-logic and so on.

It generates a written report on the issues with the “quotes” from synthetic personas:

But more importantly, it can make the necessary fixes and then re-test. Here we can see it deals with the “But I haven’t contacted support” issue that had previously caused a simulated subject to select “Neutral”:

One we feel good, the research agent can then refine the question and immediately deploy it via email, a URL or even Prolific:

You can take the survey here.

The research agent can analyze the real survey and/or simulated survey:

Under the hood, the research agent is not just drawing on context to all this: it’s writing the Python code to actually analyze the data. This matters because it is easy to analyze more data when it comes in, the research agent (or any coding agent) can reason over what was done and find issues.

Concluding thoughts

This is a toy case, and you might catch it on a careful read. The point is that a synthetic pass catches it reliably and instantly, across every question at once, for the cost of a prompt—and the harder, subtler versions of these same problems are the ones you’d otherwise only discover after fielding.

Some References

Beatty, P. C., & Willis, G. B. (2007). Research synthesis: The practice of cognitive interviewing. Public Opinion Quarterly, 71(2), 287–311. https://doi.org/10.1093/poq/nfm006

Presser, S., & Blair, J. (1994). Survey pretesting: Do different methods produce different results? Sociological Methodology, 24, 73–104.

Presser, S., Couper, M. P., Lessler, J. T., Martin, E., Martin, J., Rothgeb, J. M., & Singer, E. (2004). Methods for testing and evaluating survey questions. Public Opinion Quarterly, 68(1), 109–130. https://doi.org/10.1093/poq/nfh008

Tourangeau, R., Rips, L. J., & Rasinski, K. (2000). The psychology of survey response. Cambridge University Press.

Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage.

With EDSL, you can design forecasting questions in whatever form makes sense — from open-ended free text to tightly specified schemas — and have the responses automatically returned as a well-structured dataset. This makes it easy to experiment with different ways of framing questions for both AI agents and humans while ensuring that outputs remain standardized, comparable, and ready for analysis.

In this post, we’ll walk through a simple example using EDSL to design a forecasting question, administer it in multiple formats to an agent persona powered by an LLM, and inspect the results. We’ll also show how unstructured free-text answers can be transformed into structured data automatically. For more details on the methods used, see our documentation page.

EDSL is an open-source package for simulating surveys with AI agents and LLMs of your choice. It is maintained by Expected Parrot and available under the MIT License. Please join our Discord to ask questions, submit feature requests, and connect with other researchers!

1. Defining Forecasting Questions

Let’s start with a forecasting question about U.S. CPI inflation in December 2025. We’ll frame it two ways, using different EDSL question types that allow us to specify the format of the model’s response:

• Structured (dictionary format): where we prompt a model to provide a lower bound, an upper bound, and rationale.

• Free text: where a model can respond however it chooses.

We combine the questions in a survey to administer them together. Note that by default questions are administered async, so we can get choose whether to generate independent answers or (optionally) add within-survey memory (here we do not):

from edsl import QuestionDict, QuestionFreeText, Survey

question_text = """What will the year-over-year U.S. Consumer Price Index (CPI) inflation rate be in December 2025? Provide your forecast as a range (e.g., 2.0%-4.0%). The range should reflect the interval you believe has a 90% chance of containing the true outcome, based on BLS published data."""

q1 = QuestionDict(
    question_name = "q1",
    question_text = question_text,
    answer_keys = ["range_lower_bound", "range_upper_bound", "rationale"],
    value_types = [float, float, str],
    value_descriptions = [
        "The lower bound value of your forecasted range.",
        "The upper bound value of your forecasted range.",
        "The rationale for your forecast."
    ]
)

q2 = QuestionFreeText(
    question_name = "q2",
    question_text = question_text
)

survey = Survey(questions = [q1, q2])

2. Creating an Agent and Running the Survey

We’ll give our agent a (very) simple persona — a “professional forecaster” — and then select a model to generate the answers. EDSL works with models from many popular service providers; you can check available models, token prices and performance here. To administer the survey we simply add the agent and model, and then run the job:

from edsl import Agent, Model

agent = Agent(traits = {"persona":"professional forecaster"})

model = Model("gemini-2.5-flash", service_name = "google")

results = survey.by(agent).by(model).run()

3. Inspecting Results

The results are a formatted dataset that includes columns of information about the questions, agent, model, tokens, raw responses and other details of the job. We can inspect them at Expected Parrot or in a notebook. Here we print just the answers in a table:

We can flatten a dictionary answer for easier analysis (see methods for working with results SQL, pandas, or other tools):

3. Turning unstructured text into structured data

We can also use other EDSL question types to reformat the free text response. Here we create a "scenario" for the text in order to add it to questions prompting a model to extract the range values (as floats) and summarize the rationale in the original answer:

from edsl import Scenario, QuestionNumerical, QuestionFreeText, Survey

# use the q2 answer as a Scenario in new questions
unstructured_answer = results.select("q2").to_list()[0]

s = Scenario({"unstructured_answer":unstructured_answer})

q1 = QuestionNumerical(
    question_name = "lower",
    question_text = "Identify the LOWER bound value in the following response: {{ scenario.unstructured_answer }}"
)

q2 = QuestionNumerical(
    question_name = "upper",
    question_text = "Identify the UPPER bound value in the following response: {{ scenario.unstructured_answer }}"
)

q3 = QuestionFreeText(
    question_name = "rationale",
    question_text = "Summarize the rationale provided in the following response: {{ scenario.unstructured_answer }}"
)

survey = Survey(questions = [q1, q2, q3])

results = survey.by(s).by(model).run() # no agent used

4. Why This Matters

Forecasting often benefits from a mix of formats:

• Structured formats make it easy to aggregate, compare, and analyze forecasts.

• Free text formats capture richer reasoning and context.

By combining both in the same survey, you can explore how LLMs and human respondents differ in not just what they forecast, but how they explain their forecasts.

Try It Yourself

You can reproduce this workflow by installing EDSL:

!uv pip install edsl -q

Log in to Expected Parrot (your account comes with free credits for API calls to LLMs for getting started):

from edsl import login
login()

Download the complete notebook for the code examples above here.

Free research tools

Our open-source tools let you simulate surveys using AI agent personas and popular language models of your choice. Results are returned to you as formatted datasets specified by the question types that you choose (free text, multiple choice, linear scale, matrix, etc.), with columns of information about all the components of your survey: agent traits, model parameters, question details, answers, tokens, API costs, raw responses, and more.

The tools come with built-in methods for analyzing results, and also let you collect and compare responses from humans as needed. When you run your simulations at Expected Parrot your surveys and results are cached automatically, so you can retrieve and share them to let others reproduce and build on your work at no cost. You also get free credits for API calls to LLMs.

Some quick links:

• EDSL package at GitHub: https://github.com/expectedparrot/edsl

• EDSL documentation: https://docs.expectedparrot.com

• Create an account: https://www.expectedparrot.com/login

• Getting started: https://www.expectedparrot.com/getting-started

Getting started

The EDSL package is available at GitHub and PyPI and is free for anyone to use. Our docs page has tutorials for getting started and modifiable demo notebooks for exploring use cases and experimenting with AI agents and LLMs. You can run simulations on your own computer or create an account to run surveys at Expected Parrot and share projects and results with others.

Free credits

Your Expected Parrot account comes with $25 in credits for sending surveys to LLMs, and a unified key that provides access to all available models. You can also use your own keys for LLMs, and grant access to other users without sharing keys directly:

Credits are deducted from your account to cover API calls to LLMs using your Expected Parrot key. You can purchase more credits at your Credits page, and earn free credits by sharing your referral code. Each time someone signs up with your code and runs a first survey job we automatically add 1,000 ($10) in credits to your account:

(If you haven’t signed up yet, please feel free to use our friend Luke’s referral code :)

Teaching materials

If you’re looking for ready-to-use teaching materials:

• Our documentation page provides a starter tutorial and notebooks for a variety of use cases.

• Our blog has examples of methods and use cases too, including:

  • How to send a survey to LLMs and humans

  • Comparing model performance

  • Data labeling

• We have slides for teaching EDSL basics that we’re happy to share! Send us a message at info@expectedparrot.com or post a message at our Discord channel.

Example class project

NYU researcher James Traina recently designed an MBA case study using EDSL to simulate complex market research scenarios. The exercise puts students in the role of product strategy managers at Thunder Bank, tasked with justifying a $75 million branch transformation using AI-powered customer insights. Read what the students thought about it!

We've built features that streamline this human-AI comparison process in both our open-source EDSL package and our no-code builder. The workflow consists of: creating a survey, running it with AI agents, then sending it to human participants. Since AI and human results use identical formatting, you can analyze both datasets with the same methods.

Below we demonstrate this process with a simple survey about some unforgettable Jane Austen quotes, gathering responses from both AI agents and human participants. Note: Every step shown in code can also be accomplished through our no-code builder—reach out if you need help getting started!

Create a survey

We start by creating a survey. EDSL comes with many common question types we can choose from based on the form of the response that we want to get back from models (and humans):

Note the use of {{ scenario.placeholders }} in the questions. Scenarios allow you to create versions of your questions efficiently with different texts, PDFs, CSVs, images, videos, tables, lists, dicts, etc. Here we post some images to Coop and inspect them:

When we create scenarios we can add any extra fields for metadata that we want to include in the survey results, without having to add it to the questions directly—e.g., we did not include placeholders in the questions for “novel” and “speaker” but we can make them columns of the results that are generated:

Run with AI

Next we can design AI agents to answer the questions and select models to generate those responses. EDSL works with many popular models; you can check details on current available models here and see which ones are being used the most at our cache stats page. Here we create some simple personas and select a couple vision models to use with the survey. Note that agent traits and personas can be *much* more detailed than this—we’re working on lots of features for helping you generate them, please get in touch to test—and you can specify the model parameters (here we use defaults). We add the scenarios, agents and models to the survey when we run it:

This generates a dataset of results that we can immediately begin analyzing (see all results in the notebook and at Coop).

Run with humans

To run the same survey with humans, we start by calling the humanize method to generate a web version of the survey and a Project for collecting responses:

We can inspect and share the respondent URL:

We can create a study to launch the same survey at Prolific:

Responses are automatically added to your Project page and can be pulled into your notebook for analysis:

References

Details on methods, tutorials and example code for a variety of use cases are available at our documentation page.

A complete notebook of example code shown above is available here.

A slide deck version of the code is available here:

• Use your own keys

• Use an Expected Parrot key to access all models and participant platforms

• Mix and match keys for services as desired

How it works

Your Expected Parrot account comes with a key that lets you run surveys with all available language models and human participant platforms (Prolific). The key is automatically available and can be viewed at the “Keys” page of your account:

If you want to use your own keys for LLMs, scroll down the page to see the option to add keys. You can de/activate keys individually at any time, share access to them with team members, and set limits on their usage. Whenever a private key is activated it is used instead of your Expected Parrot key when you run jobs and this will be indicated in your results:

If you want to let other users run surveys with LLMs using your credits, you can also create temporary access tokens:

Human respondents

We recently added features for launching your LLM surveys with human respondents. These features are available whether you are working in code with EDSL or using our no-code builder (details below). You can choose whether to generate a shareable link for a web version of your survey to send to your own respondents, or you can create studies at Prolific and filter participants as desired (e.g., to align with your AI agents).

When you launch studies with Prolific you will also use your Expected Parrot key by default, and transactions will appear in your account the same as with LLM survey jobs. If you instead want to use your own key for Prolific, simply add it to your Keys page as well:

This will ensure that charges are sent to your personal account, but your results will still be available at your Expected Parrot account.

Working in code

If you’re working directly in EDSL code (in a notebook, instead of our no-code builder), you can call the humanize method to generate a shareable web version of your survey, and use Coop methods for launching Prolific studies and collecting responses. Human results are formatted the same as LLM results, so you use all the same built-in methods for analysis.

Learn more about humanize here, Prolific studies here, and analytical methods here.

Questions?

Let us know if you run into any issues! Post a message at our Discord or send an email to info@expectedparrot.com.

The Simulmatics corporation, founded in 1959, aimed to predict the future using simulations (it's right there in the company name). Those simulations could be run on newly available digital computers, informed by the micro-data that was, by then, routinely collected and digitized. Applications were wide and spanned politics, government and industry: who would turn out to vote, how a message would land in a counter-insurgency campaign, whether readers find the plot in a novel compelling, and so on. And because you could fiddle with the simulation, you could estimate the effects of different choices—not just how would this message affect counter-insurgency efforts, but what message would work best (at least in simulation world).

The appeal of having a simulated model of the world is clear. Unlike in the real world, in a simulated world, we get to see the road not taken and the road taken. We can let time run forward at any speed we want, stop it, wind it back, change something and see what happens because of our change. We can try 1,000 different messages to one simulated person and they will never tire or complain; we can modify a hypothetical product in a hundred different ways—a product that in real life takes 6 months and costs $6M to modify—and get customer reactions. We can play out a jury trial with a variety of jury compositions and arguments. Even when fully deterministic, this process of simulation often reveals insights—anyone playing SimCity or Civilizations has surely experienced this sum-is-greater-than-parts phenomena.

But Simulamatics, the company, was a failure. Jill Lepore's book on their history, If/Then: How the Simulmatics Corporation Invented the Future, has a title that's a lie—or really metaphorical. Simulmatics didn't invent the future. What she meant by "inventing the future" is that our current focus on collecting data and trying to build models to make predictions was prescient and presaged by them. Collecting enormous amounts of data and using it to make predictions is our current day practice. Yet our analytics are often a pale imitation of what a simulation could offer.

Scintillating as it may be, a table of regression coefficients does not allow us to do all the open-ended knob-turning and exploration we can with simulations. Instead, when doing even serious analytic work, we have to make unpersuasive, often shaky arguments about how data from one domain will apply to some other domain—or hope that how people respond in a pencil-whipped survey will tell us about how they would respond in real life. Or hope that the people we can cajole in taking an online survey to win a free iPad are like people CIOs buying an ERP system. Perhaps the closest we come to this "two roads at once" feature of a good simulation is the A/B test. That we do and put up with all this crap anyway speaks to how right Simulatics was about the demand. We want to know the future now—or really, we want to know the possible futures so we can pick the one that suits us.

Simulations are like maps; most analytics are trip reports

To make an analogy, a simulation is like a map. You can trace your finger on any path you like and have some sense of what that path will be like if you took it for real—steep or flat; wooded or grasslands; will I encounter a river? A mountain? Is there a pass through the range? The map will, of course, elide details. It will not tell you what kinds of flowers will be in the meadow, never mind whether they are in bloom when you hit the meadow. But you can be reasonably confident that the meadow is there, about where it is supposed to be. As such, you can use it to make plans. A map is a prediction technology and thus a planning technology. Of course, maps can be bad and maybe the meadow is not there. The "map is not the terrain" is a cliche but a cliche for a reason—it's true and important. This first map of the Americas basically gets it right but it has…some issues:

But as bad as this map is, it would actually be remarkably useful for planning, say, a voyage to California. Far better than nothing and it would lead you to good decisions; if you wanted to skip Tierra del Fuego and cross overland, better do it near Panama than Brazil. And even in 1505, I could simulate my journey (trace my finger on the map) at a speed faster than any human has traveled, and for free and without a ship. Neat trick.

Most analytics data is "Alice walked this particular path 3 months ago and here is her report on what she saw"; if our instrumentation is really top-notch, maybe we have a full video of this journey. This is…something. But we know precious little about what even a slight deviation from that path might mean and so generalizing Alice's experience to other paths we might take is a big leap. Maybe her report is "I walked across a vast desert" and in that case, we might implicitly make a map based on her report and use that for our planning. If we can do it, an A/B test is much better—we can send two scouting parties: Lewis goes the northern route (A) and Clark goes the southern route (B). If we can do this, it's great but we do only get two routes. This is of course, expensive and just gives us A and B, but at least we can make comparisons.

The main solution—explicit in the social sciences and rarely made explicit in business—is theory-building. We hope that with some general social science theories about how humans behave, we can use that theory to make predictions in new settings. Practically-minded businesses never describe themselves as theory building (too pretentious and too academic) but that is precisely what they are doing—creating theories about the things they care about that they hope will generalize. What is "know your customer" if not "building an implicit model of our customer so you can predict what they will do in new situations"?

Of course, social science theories face this tension between usefulness and parsimony and so they end up with theories not really up for prediction, at least compared to the natural sciences. There are a lot of reasons of course—one is simply that it's hard to incorporate all the factors that matter. But perhaps the most fundamental is that people are complex. They are nuanced, strategic thinkers. They reason about how others will react, which affects how they will react, which in turn affects the whole system. They think about the future and the past. They have memories and emotions. They can make plans and enlist others, using their mental model of others to help them coordinate. It's all great stuff but it makes predicting their aggregate behavior hard. What you need, for many applications, is a model of humans in your environment that is nearly as complex and subtle as actual humans.

Back to Simulatics: It was supply, not demand

Simulmatics failed not because there was no demand for what they proposed selling: they failed because they were about 65 years too early, technology-wise. They were founded in 1959 but what they wanted to do you can only do credibly in 2025. To make open-ended social simulations that actually work well and are very general, you need a reasonable simulacra of actual individual human behavior that can be plugged in: how we think, how we reason, how respond emotionally and rationally to stimuli. And we need these simulated humans to themselves generate stimuli that other simulated humans can respond to, and so on. This is just now possible with LLMs. They are, of course, not actual humans—but they are probably close enough for useful map-building. In a way, this is what economics does, with great success: we plug in a narrow model of humans (one that can do calculus when making decisions) and then "simulate" not numerically but by solving the equations implied by our set-up.

What we're doing: Tools for explorers, not making maps

Expected Parrot is not Simulmatics, in several important ways. Perhaps most fundamentally, Expected Parrot is not selling answers—we are making the tools that will let others find answers for the questions they care about. By creating open-source tools and a platform for sharing work, people can collectively explore what is possible with this technology, building on the work of others—which includes critiquing the work of others ("That's not where California is!").

Even with the potential for building these kinds of simulations, there is no guarantee that any particular simulation will be good or accurate—it depends greatly on how the simulations are designed, what data agents have, how they interact, and so on. And there are surely questions for which this approach does not work. But this "tools, not answers" is completely standard for powerful tools: Microsoft will not promise you the spreadsheet you make is good or will give you the correct answer—their commitment is to give you Excel, and that you can trust Excel to do what Excel claims to do—and then it is on you to use it appropriately.

Our vision is that our tools and services are so easy to use—but also so powerful—that the first question before any decision is "How can we simulate this?" and the answer is usually very clear and not much work: it is trivial to build agent pools, design studies, launch simulations, analyze results and plan follow-on experiments. And, critically, this is all done while it is always made clear what the gap is between the simulation and what we know about the real world, as best we can tell.

As a team, we are focused on very general tools because we believe that there is enough commonality across all would-be simulations that the tools can be "horizontal" without losing their power: that the same tools that would allow an education researcher to explore how different lesson plans affect learning could also be used for a product designer coming up with a new kind of yogurt packaging. We don't need "desert mapping tools" and "grasslands mapping tools" per se; a transit is a transit no matter what we use it for. While the domains are radically different, the fundamental object of inquiry is no different—what humans will do? We think there will be and can be a kind common core. Rather than homogenize, common tools will facilitate cross-fertilization, mixed methods and boundary-spanning work.

Future

As AI advances, we expect that researchers will work at higher levels of abstraction. Now we design research instruments; tomorrow we'll propose research questions and the AI will propose surveys; next week the AI will be suggesting what research questions we should even be asking and doing all the associated research. Our tools will automatically construct the right agents, run the right simulations, do the right analyses and present the key results—perhaps all within seconds. And as our knowledge of how to combine humans and AI answers improves (an active area of research), we'll get precise, automatically generated predictive models that optimally make use of both sources of information. One could think of it as an infinitely scrolling map (but flawed, like all maps) that gets built on the fly so quickly that it seems like the full map already exists. It's going to be a wild world and we're excited about helping to build it.

• EDSL, an open-source Python library for simulating surveys and experiments with AI agents and LLMs (read our docs); and

• Coop, an integrated platform for conducting hybrid AI/human research interactively (login/signup).

The team had heard about our tools and thought they could be helpful for gathering information about potential customers for startup ideas. But the presentation went sideways when I brought up a slide of EDSL code (and accompanying snapshot of our interactive survey builder tool at Coop) to show how to begin a research experiment-as-a-survey for LLMs:

The same first step at Coop (you can click “Code” to get the EDSL code, for technical and non-technical teams to collaborate):

The team was insistent that early-stage founders, particularly non-technical ones, don’t have time to learn a new tool for time-sensitive market research. In fairness, you can obviously get a ton of fast research done with user-friendly AI chat interfaces. But if you want to do any pressure testing of the answers that you’re getting from LLMs—What happens when I emphasize this part of the customer persona instead? What happens when I modify the description of the product, or rephrase the question, or try a different model, or provide more information? etc.—it quickly becomes unwieldy to collect data and analyze feedback and ideas via unstructured text.

It also makes it hard to compare your AI results to any information that you’re able to collect from your actual target audience. In order to compare your LLM and human answers you’ll want a dataset—even a simple spreadsheet—with organized information about the personas that you included in your LLM prompts, the information that you have about your human respondents, and the answers that each of them provided to your questions. Your dataset should also show all the variations of the prompts that you sent to LLMs, which is only possible if you…

Tinker with your prompts!

It’s not necessarily obvious what information you have—or do not have—about your target audience is relevant to the questions that you want to ask, or how its presentation in LLM prompts is impacting the responses. If you want to increase your confidence in your AI results you should tinker with your questions and AI personas and include humans in-the-loop when possible. EDSL is designed to make all of these things easy in a number of ways:

• Explore question types—EDSL has many common question types (free text, multiple choice, etc.) to explore the impact of the presentation of a question on responses.

• Test variants of your prompts—EDSL has many methods for auto-importing data to efficiently run multiple versions of your questions, instructions and agent personas all at once.

• Compare models—EDSL works with many popular service providers, and your account comes with a key for accessing all of them at once to easily compare performance on your tasks.

• Validate with humans—EDSL provides methods for launching your surveys with humans to compare results and improve your workflows. Choose whether to send surveys to your own respondents or use our Prolific integration.

• Analytical tools—Results—LLM and human—are returned as formatted datasets containing details about questions, agents, models, responses and costs that you can immediately begin analyzing with build-in methods, or export.

EDSL is easy to learn.

No, you do not need to be any kind of coding expert or Python wiz to master EDSL. Our docs have demo notebooks for a variety of use cases that you can download and modify to get started. You can also use Coop to create a project interactively and then get the EDSL code for it (click the “Code” button in the snapshot above), or use our GPT for generating EDSL code. The concept—design an experiment as a survey answered by AI personas using LLMs to generate the answers—and the base components are straightforward:

• Construct questions (free text, multiple choice, linear scale, matrix, etc.)

• Combine them in a survey and add any desired logic (e.g., skip/stop rules)

• Optionally design personas for AI agents to answer the questions

• Send the survey and agent personas to a language model to generate the responses

• Get the results back in a formatted dataset to analyze with built-in methods

Here’s a notebook with a quick example. I start by constructing a survey and AI persona and select a model to generate the responses:

Next I launch a web version of the survey answer it (at the respondent URL):

I can pull the results back into my workspace to compare with the LLM results:

… and also inspect them at my Coop account:

Market research & other use cases

If I can convince you to give it a try, here are some ways to use EDSL to conduct market research:

• Simulate customer personas: Design Agent objects with a variety of traits and preferences to represent different customer segments.

• Test messaging and branding: Use question types such as QuestionFreeText or QuestionMultipleChoice to get feedback on product descriptions or branding materials.

• Compare product features: Create Scenarios for different feature sets and use QuestionRank to have the agents rank preferences or interest.

• Evaluate price sensitivity: Present different pricing scenarios and ask agents to choose or rate affordability using QuestionLinearScale.

• Analyze perceived value: Use open-ended questions to elicit reasons behind preferences or concerns with QuestionFreeText.

• Explore distribution channels: Pose Scenarios about online vs. in-person availability and ask which they prefer and why.

• Gauge emotional response: Include qualitative questions with QuestionFreeText to measure reactions to marketing copy, visual content, or product pitches.

• Test variations across contexts: Vary Scenario and Agent attributes to explore how preferences change by traits or product conditions.

• Benchmark against competitors: Create Surveys comparing your product with existing ones and assess perceived strengths and weaknesses.

If you have any questions about using our tools for your research, please get in touch! info@expectedparrot.com

1. Create & run a survey in EDSL

Start by constructing questions in EDSL, our open-source package for running surveys and experiments with AI agents and LLMs. Choose from many common question types based on the form of the response that you want to get back from a model. You can optionally design personas for AI agents to answer the questions, and specify which of many popular LLMs you want to use to generate the responses. Run the survey by adding the agents and models to the survey and calling the run() method. This generates a formatted dataset of results that you can analyze with built-in methods:

When you run your survey at Expected Parrot your results are automatically stored at your Coop account where you can access and share them. You can check a progress report while the survey is running, and see details on costs for each response when it is done:

2. Inspect results

Results include information each component of the survey job: questions, prompts, agents, models, raw responses, costs, etc. You can inspect results at your Coop account, and also use methods for analyzing them at your workspace. Here we select the responses, and the comments that are automatically added to them (learn more about modifying user and system prompts for your specific research needs):

3. Generate a web-based version of the survey

To collect actual human responses to your survey, call the humanize() method on the Survey object to generate a web-based version of it. You’ll get a link that can be shared with anyone you want to answer the survey, and another link for accessing responses at your Coop account (the admin_url):

You can also use our interactive survey builder tool to construct a different web-based survey, or to edit the one you generated from your EDSL survey (e.g., if you want to add some screener questions for your real respondents).

4. View & share the survey URL

Here I go ahead and answer the survey myself, as my response can offer a reliable check on the AI agent’s from above:

No problem!

5. Combine & analyze results

LLM and human results are datasets that you can analyze with built-in methods for working with results (e.g., to use them as inputs for follow-on questions for LLMs). You can also export them, e.g., as dataframes:

We’re always adding new methods like these — please send us your feature requests! Send us an email (info@expectedparrot.com), post a message at our Discord or DM us on X.

Below we share some quick examples of methods for retrieving multiple sets of results at once, combining them, and analyzing them with SQL queries. This can be useful when you are re-running an experiment with different AI agents and/or language models, or want to compare fresh responses over time.

Retrieving saved results

When you run an EDSL survey, details about the survey job and results are displayed in your workspace while the job is running (you can also hide this output by passing verbose=False):

When the job is completed the results are automatically stored at your Coop account, where you can view, organize, share and export them (a link to Results is in the job status details shown above):

You can retrieve stored results at any time by passing the unique identifier to the pull method:

You can also give the results a unique alias, if desired:

This allows you to retrieve by alias instead:

Multiple results

If you want to retrieve several sets of results at once you can use the list method to identify and select them. For example, say you reran the survey above multiple times with different models:

You can then specify which results to retrieve, e.g., the 3 most recent:

The page specifies which set of 10 objects to look back to (1 by default, the most recent page) and page_size specifies how many objects to return (10 by default). If you want to search results by survey job status or description instead you can call the method on Jobs (all parameters are optional):

Once you’ve identified the relevant jobs or results you can use the fetch methods for instantiating the results. These are equivalent:

This returns a list of results objects. You can combine them to analyze them together (the columns must be identical):

Methods for analyzing results

Now all of the methods for working with results are available to us. For example, you can use the sql method to run queries on the combined results:

If you want to share an analysis, or a notebook of code for reproducing it, you can post that to Coop as well:

Please see our docs for more examples of analytical methods and instructions for getting started. If you don’t see a method that you need, please post a message at our Discord channel or send us an email to request it.

Question: How many Turing Award winners graduated from MIT?

Say I want to know how many Turing Award winners received a degree from MIT. This is a very answerable question, but there very likely isn’t an exact dataset I need with an “mit_degree” boolean field. I’ll need to construct it.

In the distant past (say, 3 years ago), I could have taken Wikipedia bios and maybe done a regular expression search for “MIT”—though given how many winners are affiliated with MIT but did not graduate, this would be error-prone. Ultimately, I’d probably have to review by hand. Now, of course, we can use AI.

First, I’ll show approaches that don’t really work (with Claude) and that do work (with o3) but also a method that works with less sophisticated models and that has a number of other advantages as an answering approach.

Claude gives an MIT ‘vibes’ answer

The Claude answer definitely finds some related info but includes people with MIT connections but not actual degrees. This is probably because every time someone wins a big award, universities try to establish some connection (“Herb Simon once ate in our cafeteria!”). There is also a counting effort, but garbage-in, garbage out.

o3 seemingly nails it

I asked OpenAI’s o3 “How many Turing award winners received their PhD from MIT?” and then asked about degrees generally. Watching the reasoning traces as they flashed on the screen, it seemed to do something sensible and probably what one of us would do if given the task—it got a list and then kept doing web searches until it found the relevant information, almost all from Wikipedia:

As far as I can tell, this is correct. But this was PhDs—let me instead do graduate degrees, which is what I mean to ask:

Ok, 6.

But even o3’s answer has some problems

The procedure with which this answer was obtained is opaque.

• I don’t really know the process by which these determinations were made.

• I don’t know if any of these are hallucinations (and I can’t easily try it with other models) or omissions.

• If someone wanted to reproduce my analysis, they would have to re-run this (rather expensive) inference (2m 22s).

And what if my dataset were *a lot* bigger and not publicly accessible. There are only a fairly small number of Turing winners and the list can be found in numerous places. What if I want to do something similar but with non-public data of much larger scale—say look for who graduated from MIT from my 10,000 employees in a CSV sheet?

The E[🦜] Approach

First, the data on winners is easy to get, already arranged in a nice table at Wikipedia:

To actually extract this, EDSL (Expected Parrot Domain Specific Language) has some tools for working with Wikipedia data. The basic idea is turn it into a “ScenarioList” which is, well, a list of Scenarios for a language model to assess in some way.

The data is puled down using “from_source” method. EDSL uses a fluent interface throughout and so we can chain together methods. In particular, the keys need to be valid python identifiers, which is what that “give_valid_names()” function does. There is also an “augment_with_wikipedia” function that can get get more data about a particular search query—-in this case, the prize winner.

Here is what one entry looks like:

Note that I have a field that has the full biography from Wikipedia. I can then use that to ask a question: Did this person receive a degree from MIT? For this, we use a yes-no question type. Note the use of a jinja2 templating syntax:

The last line uses a fluent interface to combine this question with the scenario list and then “run” it with a language model. (The current default model in EDSL is gpt-4o; you can easily specify a different model, or run it with several models at once to compare responses.) We can see that answering question takes about 3 cents’ worth of tokens:

If I—or anyone else—were to run it again, the responses would be returned at no cost, as they have been “cached” at E[P]. With our results object, we can now tally up the results:

Oh, wow! But wait—boy genius o3 is wrong!

Note that o3 said 6, but the EDSL approach got us 7. NB: This wasn’t a big set-up to show our tools work better. I just noticed this as I was working it out. What’s going on? Forgetting someone?

Turning the knife:

You can view this and other metrics for the Universal Remote Cache at our cache stats page:

This page is updated daily to show total cache entries and total savings (costs avoided whenever a stored response was retrieved instead of resending an API call). The information is available for each language model used, so you can also tell at a glance which models are popular for research projects.

What is the Universal Remote Cache?

The Universal Remote Cache (“URC”) is a collection of all the unique prompts (user prompts and system prompts) that have been sent to language models via Expected Parrot, together with all the responses that were returned. It is a shared resource that is available to all users for free whenever you run a survey at Expected Parrot.

How it works

When you run a survey at Expected Parrot your results draw from the URC by default. This means that if your survey includes any prompts that have been run before—by you or another user—the stored responses to those prompts are retrieved from the URC and included in your results, at no cost to you. A new response is generated for each of your prompts that has not been run before, and added to the URC. If you want to generate all fresh responses instead you can turn this default off when you run your survey.

Your prompts and responses are shown at the Cache page of your account; details on costs (and costs avoided) in generating them are available at your Jobs and Transactions pages. To show how it works, below we run a simple question with some models and check the job, transactions and cache details. Then we re-run it and verify that the cached responses were retrieved and no credits were used.

Running a survey in notebook:

The survey job is added to the Jobs page with information about credits used to generate the results:

When we re-run the question we can see that no credits were consumed for the second job:

At the Cache page we can verify that 3 entries were created and then retrieved:

We can generate fresh responses by passing a parameter fresh=True to the run() method:

We can see that credits were consumed again to generate the new responses:

The cache has new entries (fresh unique responses to existing prompts are automatically indexed):

How remote inference works

This caching of prompts and responses happens automatically whenever you run a survey using EDSL. The URC is used whenever you run a survey “remotely” at the Expected Parrot server; when you run a survey “locally” on your own computer your prompts and responses are cached on your computer instead.

To activate remote inference, navigate to the Settings page of your account and confirm that the remote inference option is toggled on. This allows you to run your surveys at the Expected Parrot server using your own API keys for LLMs and/or an Expected Parrot key. Learn more about how remote inference works.

Features

Features of the Universal Remote Cache include:

• Free access: The URC is free to use and available to all users, regardless of whether you are running surveys at Expected Parrot with your own keys for language models or an Expected Parrot key.

• Free storage & retrieval: There is no limit on the number of responses that you can add to the URC or retrieve from it.

• Automatic updates: The URC is automatically updated whenever a survey is run at Expected Parrot.

• Multiple responses: If a fresh response is generated, it is automatically added to the URC as well, with an iteration index.

• No deletions: You cannot delete entries from the URC.

• Only verified responses: You cannot manually add entries to the URC. The only way to add responses is by running a survey at the Expected Parrot server.

• Privacy: The URC is not queryable, and no user information is available.

• Sharing & reproducibility: A cache for a survey is automatically attached to each results object, which can be shared with other users. A green checkmark at Coop indicates to other users that your results are verified and can be retrieved from the URC for free:

  

  Note: The URC is not available for local inference (surveys run on your own machine).

Frequently asked questions

How do I add responses to the URC? This happens automatically when you run a survey at the Expected Parrot server.

How do I retrieve a stored response? When you run a question you will retrieve a stored response by default if it exists. If you want to generate a fresh response you can use run(fresh=True).

How do I know whether I will retrieve a stored response? Results that were generated at the Expected Parrot server will show a verified checkmark ✓ at Coop. If you rerun a survey with verified results you will retrieve stored responses by default.

Is the URC queryable? Can I check whether there is a stored response for a question? No, the URC is not queryable. You will only know that a response is available by rerunning surveys with verified results.

Can I see which user generated a stored response in the URC? No, there is no user information in the URC.

Can I add my local cache to the URC? No, this is not allowed. The purpose of the URC is to provide a canonical, verified collection of responses to allow researchers to be confident in results and easily reproduce them at no cost. By only allowing responses generated at the Expected Parrot server we can verify the results that are reproduced.

What if I want to run a survey remotely but do not want my responses added to the URC? This is not allowed. Any new or fresh responses generated at the Expected Parrot server are automatically added to the URC. If you do not want to add responses to the URC you must run your surveys locally.

Can I access the URC when I run a survey locally? No, the URC is only available when running a survey remotely. However, you can pull entries from your own cache (responses that you generated or retrieved from the URC) to use them locally at any time.

Can I delete a response in the URC? No, this is not allowed.

What happens if I delete my account? Any remote cache entries that you generated will remain in the URC. All other account information will be deleted.

Please send us other questions about how it works!

When you run a research project with Expected Parrot it’s easy to track the details of your LLM costs and usage, using built-in methods at your workspace and at your account at Coop, a platform for creating and sharing LLM-based research. You can also replicate and share your results for free. Below we highlight some new and key features for easily and precisely tracking your costs.

Before you run an experiment…

When you’re developing an LLM research project you want to know (i) the token rates for the models that you plan to use and (ii) estimated costs for your prompts and the responses that you will receive back from the models. You also want to avoid making duplicative API calls to LLMs (resending identical prompts when you do not expect the response to change) to eliminate unnecessary $$ charges.

Checking model prices & capabilities

It can be time-consuming to gather token rates from various model service providers’ webpages, and to determine which models are capable and appropriate for your research goals and requirements (e.g., do you need a vision model for some of your prompts?).

This is why we created a model pricing and performance page where you can quickly check current token rates for models of all available service providers, and their recent performance on test questions:

The page is updated daily with details on the test questions that were used with each model. We are adding test questions that use videos! See example code for using videos with your questions.

Estimating costs

Once you’ve drafted your user and system prompts (i.e., created your questions and any agents), you can use built-in methods for estimating the costs of sending them to models (input tokens) and the costs of the responses that you will get back from the models (output tokens).

For example, here we create a simple question and then use a method for estimating input and output token costs of sending it to two different models. We can see the details also include the total credits that will be placed on hold while the survey job is running and then released when the actual cost is determined:

After a survey is run, we can verify the actual token costs for each model and question in the dataset of results that is generated:

You can also view token and cost details at your Coop account. Your Jobs page shows the final cost in credits of API calls to each model used with the survey:

Your (new!) Transactions page shows additional details about credits placed on hold based on job cost estimates together with the final cost details:

Avoiding unnecessary API calls

A key feature of EDSL is automated caching of prompts sent to models and responses received from them. This ability to retrieve responses to questions that have previously been run at no cost means that you can replicate and share any research created in EDSL for free.

For example, if we rerun the example question from above we can confirm that no additional costs are incurred:

This can be useful when iterating on a research project and when publishing your research results for others to review and replicate.

• Unified access: Connect to multiple language model providers with a single API key and universal methods—no software engineering required

• Structured data collection: Automatically formatted datasets of responses eliminate manual data cleaning work

• Analysis tools: Built-in methods for analyzing, reproducing and exporting results

How it works

EDSL is an open-source Python package designed to simplify conducting experiments with AI agents and language models. A typical workflow consists of the following steps:

• Create questions (free text, multiple choice, numerical, matrix, etc.)

• Combine questions in surveys with custom logic (skip patterns, stop rules, etc.)

• (Optional) Create personas for AI agents to answer the survey

• Select language models to generate the responses

• Analyze results in specified datasets

Simplified access to models

EDSL is designed for researchers who want to work with language models without spending lots of time on software development. It eliminates the need for one-off coding required to access individual service providers by letting you connect to them all in the same straightforward way.

In EDSL, you can access models individually or collectively by simply selecting them and specifying desired parameters. For example, here we create a list of OpenAI models in order to use them all simultaneously with a survey when we run it. We can inspect the default parameters for the models that will be used:

You can also create multiple instances of the same model with different parameters. Here we create a new model list where we adjust the temperature of a single model:

Results as formatted datasets

Running a survey with multiple models at once allows us to readily compare responses in the dataset of results that is generated. Here we run a simple survey (consisting of a linear scale question and a numerical question—see examples of all question types) and inspect the models’ responses:

These results can be viewed at Coop as well:

Other columns of the results include:

• User and system prompts

• Question details

• Agent traits and instructions

• Model temperature and other settings

• Raw and formatted responses

• Log probs

• Input and output tokens

• Costs

Works with many models

EDSL works with many other popular language models as well. You can check available models, current token prices and daily performance on test questions here:

Please get in touch to request other service providers that you want to use!

Getting started

Our documentation page includes tutorials and demo notebooks for getting started using EDSL with language models, including examples of methods for evaluating model responses.

If you have a use case you don’t see, please see us a message and we’ll create a notebook for you!

1. Open source features. Our platform is built on EDSL, our open source package for conducting research with AI agents and language models.

2. Designed for researchers. Our tools are designed for social science and market researchers who want to conduct complex tasks and experiments with LLMs—and not spend lots of time on software development.

3. Easy access to models. Our tools work with many LLMs from popular service providers that you can connect to with your own API keys or a single key managed by Expected Parrot. You can check current performance and token prices of available models here.

4. Reproducibility. Your LLM prompts and responses are stored automatically, and always available to you. Anyone else can also retrieve them for free whenever you share your experiments and results.

5. Transparency. Our tools allow you to precisely specify and control all components of your experiments. Results are generated as readily usable, formatted datasets.

6. Built-in tools for analysis. Our tools come with built-in methods for analyzing, visualizing and exporting your results.

7. Free storage. Your account comes with free storage for your projects so you can access them from anywhere.

8. Free sharing. You can choose whether to share projects and results privately, or post them publicly.

9. Free credits for API calls. Your account comes with $25 in credits for API calls to language models for getting started. We also have lots of free credits for students and research projects!

How it works

Expected Parrot tools are designed to make it easy to run complex experiments with LLMs, and to share and replicate research. A typical workflow consists of the following steps:

• Design an experiment in EDSL, our open source Python package.

• Run your EDSL code in a notebook (e.g., Jupyter, VS Code or Colab).

• View your results at your account (Coop is our free platform for storing and sharing content you create in EDSL).

• Iterate on your work using built-in methods for analysis.

• Publish your work to allow others to replicate your research at no cost.

Technical setup

Technical setup is quick!

1. Create an account with an email address.

2. Install the EDSL package on your computer. See instructions.

Explore demo notebooks

If your experiment idea is nascent, take a look at our many demo notebooks of example code for a variety of use cases. These are designed to help you get familiar with how EDSL works and also provide a modifiable template as your starting point.

Try our GPT

Not sure how your idea would work in EDSL? Try describing it to our GPT for generating EDSL code. Here’s a quick example:

Copy the code to a notebook and run it:

You can view the results at Coop:

Please get in touch if you have any questions or feature requests!

Post a message at our Discord or send us an email at info@expectedparrot.com.