tl;dr: I show how you can use Expected Parrot (E[P]) to explore a real business question faced by CVS—How do we encourage the adoption of lower cost biologics?—and make headway with a combination of AI approaches and human research. We find digital twins work well out of the box but are suspiciously under-dispersed in their answers; we run a real human study and then use that insight to build better digital twins, ready for re-use.
The E[P] workflows are inspectable, reproducible and easy to extend:
A recently WSJ article has an interesting discussion about how CVS is using “digital twins”:
Twins created with Simile are currently helping CVS Health make decisions more quickly and confidently, executives said as part of the same discussion.
What kind of decision?
When the pharmacy and insurance company wanted to encourage adoption of lower-cost biologic drug alternatives, twins of the consumer pool suggested that people wanted to hear recommendations from their healthcare providers…
Digital twins said they trust their health care providers
While the article is light on details, a main conclusion was “twins of the consumer pool suggested that people wanted to hear recommendations from their healthcare providers, said Tilak Mandadi, executive vice president of ventures and chief experience and technology officer at CVS Health.”
This seems sensible. So sensible in fact that one might thing an out of the box model might do well on this task. I gave our Research Agent a link to the article and asked it to replicate the described study:
Out-of-the-box models get the “people want to hear from their doctors”
The Research Agent asked some clarifying questions, then wrote the code to run this simulation: it created synthetic personas of both providers and patients, asking me for guidance along the way. It ran the simulation, wrote a report and then did an internal quality control check. Under the hood, the study is represented in EDSL, Expected Parrot’s open-source Python framework, so the questions, personas, prompts, models, and results are all inspectable and reusable.
Here’s a snippet of the report finding the same headline finding:
But I was a little skeptical of two things. 1) The Research Agent had access to the findings which may have influenced the twin creation and 2) unanimity is always a worrisome sign of mode collapse. So I 1) started a new session with the research question and context but without the WSJ link and 2) asked the research agent to design and run a study on Prolific based on what we asked and learned from simulated patients. With knowledge of what our pilot showed, it then helped me design a thoughtful study for real humans:
For 1), it didn’t make a difference—the finding was the same. But for 2) about 20 minutes later, we got a comparison report.
Real-human answers from Prolific
The Prolific study (only n = 20) already shows something quite interesting. Despite also having the same mode response, there’s quite a bit more diversity with a sizable number wanting to “do their own research.” This could, of course, be noise, a feature of convenience sample, etc., but it does indicate there’s more diversity. The cool thing though is you can then try to use this new human data to create better digital twins!
We have new digital twins!
With our real human data, we can do something better than just saying “ah, well there’s a gap.” We can try to close it. We have developed a companion open-source package that gives a more rigorous way to assess proposed twins when you have micro-data: take a survey with N questions, hold out all but 1 and build twins from the N-1 remaining questions, then see how well it does on the hold-out. Here’s the main take-away:
This doesn’t help with truly new questions, but it’s a sanity check. Doing this, we can see a) all frontier models do about the same and b) they do way better than a baseline (fitting xgboost using the embeddings—more on this in another blog post).
