How we read hundreds of saved AI answers without asking again
An extraction reads the responses a run already saved and turns them into rows you can count. Why we built them, the five kinds, and what reading 559 answers cost next to asking for them.
A run can leave you with hundreds of answers. One of our recent research runs asked 40 prompts of 15 combinations of model and search, and 559 of the 600 came back with an answer. Most of them were several paragraphs long with a list of sources underneath. We did not read 559 of those carefully, and we doubt anyone would.
What we wanted from that run was narrower than the answers. We were testing a marketing claim (that AI assistants drop a product when different websites describe it differently), so for each answer we wanted to know whether it reported an actual study or just advice, which sources it leaned on, and whether it supported the claim or pushed back. Extractions are how we pulled that out of the saved text, as rows we could count and filter.
Three ways to get structure out of a run
Once a run has finished, you can read everything yourself. That’s fine for twenty answers and falls apart well before five hundred.
You can ask again, this time with a format instruction: “answer as JSON with these fields.” That pays for the whole run a second time and gives you new answers instead of structure for the ones you have. It can also change what you were measuring, because a model asked to fill in a form may answer differently from one asked the way a person would ask.
Or you can keep the answers as they are and have something else read them. That’s an extraction. It reads the stored responses of a finished run and writes what it found beside them. It never edits a response and never sends the original prompt again.
We keep asking and reading apart because only reading can be repeated. Ask a model the same prompt twice and you get two different answers, so a second ask is new evidence, and it costs what the first one did. A reading can be run again with new instructions as often as you like while the answer underneath stays exactly as it was. That means you can decide what to look for after the run, which is usually when you finally know what to look for.
What reading costs next to asking
Here is that research run, and the two readings we did of it afterwards.
| What we did | Responses covered | Cost |
|---|---|---|
| Asked 40 prompts of 15 combinations | 559 answered of 600 | $43.01 |
| Jev, three classification questions per answer | 559 read | $0.07 |
| GPT-5.6 Luna, up to three evidence records per answer, reading sources too | 501 read, 58 failed | $1.21 |
The Jev reading cost about one six-hundredth of the run. The detailed reading by a full language model cost under 3% of it. If we change our minds about the questions, we can run both again for roughly the same amounts, and the $43 never has to be spent twice.
Two kinds of extraction cost nothing at all, because word match and pattern don’t call a model. On a store of a few thousand responses, that’s the difference between trying an idea on a whim and budgeting for it.
The failures in that table are real, and we left them in. 57 of Luna’s 58 failures ran into the 2,000-token limit on an extraction’s output, and one timed out. Our first attempt at the detailed reading, with Claude Haiku 4.5, never produced a row: the provider rejected the output schema for every answer, so it cost nothing and stayed on the run labelled as failed. A failed reading stays a failed row instead of becoming an empty value that looks like “this answer mentioned nothing.”
One row for every response
Every extraction writes one row per response in the run, including responses that never produced an answer, so the run is always the denominator. Each row has a status: read, pending, failed (with the reason), no text when the response had none, or not searched when Sources meets an answer that never searched. Only a read row carries a value.
“12 answers mentioned it” means little until you know whether that’s 12 of 40 or 12 of the 25 that could be read. Rows also keep their model and search mode, and the Extractions tab summarises each combination separately. We don’t average one model’s results into another’s, because the differences between them are usually the finding.
The five extractions
Word match
You give it a list of names, each with its aliases, and it finds them in every response as whole words, in any case. It is instant and free. Use it for brands, products, competitors, people: anything you can name before you start. Change the list and run it again whenever you like.
It only finds what you list. It can’t tell you about a product you didn’t know to look for, which is what the model readings are for.
Pattern
A regular expression run over each response, with case-insensitive, multiline and dot-matches-newline flags and a cap on how many matches to keep per response (up to 100). It suits anything with a shape: prices, dates, URLs, version numbers, phone numbers.
Patterns use RE2 syntax and run on a linear-time matcher, so no pattern can hang on a long answer. Back-references and look-around are refused with a message saying so, the same way RE2 refuses them. Free, like word match.
LLM Model
You pick any model in the catalog, write instructions, and choose whether it reads only the answer’s text or the text together with its sources. If the run asked follow-ups, the reader can see those too, in order, after the answer.
Without a shape, the reply lands as plain text in one column. With a shape (up to 20 fields, and one level of nested records), the model replies in JSON, and we check every reply against the shape before keeping it: unknown keys are dropped, overlong text and lists are cut, and a value of the wrong type is refused with the path to it. The evidence reading above used a shape like this, with fields for the source, the study design, what was measured and the direction of the result.
The reader gets each response inside a block marked untrusted, with no tools and no web access. Whatever a web page said to the model that wrote the answer has no way to instruct the model that reads it, and the reader’s output is stored as data.
It costs one model call per response, billed to your own OpenRouter account. You get a quote first, based on each response’s length plus your instructions at the model’s catalog price, and nothing runs until you confirm. The run’s spending ceiling covers extractions as well as the original answers. We’d pick a reader that isn’t among the models you tested, so no model ends up grading its own answers.
Jev
Jev is a decision model from TypeSafe, served through OpenRouter. You give it a context and up to ten typed questions, and it answers each one with numbers instead of prose. A yes-or-no question gets the probability of yes. A one-of question gets the likeliest of your options, its confidence, and the probability of every option. A scale question gets a position on an ordered scale you define, lowest first.
All the questions go in one call per response. Jev bills input tokens only, which is how three questions over 559 answers came to seven cents. On our run the questions were the best kind of evidence each answer reported (from a controlled experiment down to an anecdote, or none), which outcome that evidence measured, and what the answer concluded about the claim.
The Extractions tab summarises each question per combination, with the number of answers read next to every figure. “Leaned yes” means a probability of 50% or more, but the download keeps every probability, so you can set your own threshold depending on which kind of mistake costs you more. Those probabilities are Jev’s confidence in its classification of an answer. On our run they said nothing about whether the marketing claim was true, and we used them to decide which answers deserved a careful read by a person.
Sources
Sources is the one extraction nobody has to create. Every run in which some combination searched the web gets it automatically. For built-in web search it lists the citations the provider returned. For external web search it lists the queries the model ran and the pages it opened through our search tools, and each row says which of the two it came from.
A page the model opened was seen, not necessarily relied on, and a citation tells you where the model pointed, not whether the page says what the answer claims. A response with no web search has no sources and shows as not searched, never as zero sources. Sources is free, since it’s read from what the run already recorded.
Reading a large store of answers
On the research run, we started with Sources because it was already there, used Jev to sort the answers by the kind of evidence they reported, and ran the detailed reading to pull out the studies themselves. Then we checked selected sources against the original pages ourselves. Word match and pattern fit in wherever a name or a shape is enough.
Every extraction can be downloaded as CSV or JSON, one row per response or one row per record or source, with the response id on each row so you can open the original answer. An agent connected over MCP can do all of this too: list a run’s extractions, run a new one, and page through its results. A model or Jev reading only quotes until the agent confirms it.
Our market research post shows an earlier version of this workflow: we pulled product names out of 456 saved answers, then changed our alias rules and rebuilt the tables without a single new model call.
If you have a pile of saved model answers and a question you’d like to ask of all of them, write to us at contact@runanymodel.com.