What does an AI verifier actually verify?

A post on X described 300 AI agents researching company profiles, with a cheaper model checking their work. It claimed verification in roughly 100 milliseconds and zero errors after three passes.

The replies asked questions that deserve an experiment: What does the checker verify? How much does checking cost? What happens when the checker is wrong?

We read 148 non-ad posts and replies across the conversation and one nested discussion. This was a sample from X’s relevance ordering, not a survey or a complete collection of the replies. The accounts’ reports are leads to investigate, not independently verified results.

There was also a problem with the supporting material. The original post linked to a guide to organizing Markdown folders. That guide explicitly says “No Obsidian” and does not document the claimed 300-agent experiment. We could not substantiate the performance claims from those materials.

The discussion still gives us a useful starting point: what would someone need to measure before trusting this workflow with their own research?

The checker needs a specific job.

One commenter asked what the system actually verified. The author answered that it checked each claim against the original source data, rejecting an output and continuing the loop when the cheaper model did not confirm a match.

That describes a procedure. To evaluate it, we would still need the source, the claim, the acceptance rule, and examples of mistakes it catches and misses.

Consider a fictional company profile. A source published in 2022 says the company has 40 employees. An agent writes that the company has 40 employees today. A check that the number appears in the source asks a different question from a check that the source supports a current headcount. Neither establishes whether the original source was accurate.

Before choosing the checker, define the decision. For this experiment, we would start with a narrow one: does the supplied evidence support this particular answer, including its date, entity, and qualifications? An answer with insufficient evidence should be allowed to remain unresolved.

Choosing the verifier is an empirical question.

Tyson Maly questioned the expensive-writer, cheap-verifier arrangement: “I have not had good results using the cheaper model to verify.” Another participant described using a different model and harness for verification.

Those are useful experiences to follow up on. They do not establish that a cheaper verifier always fails, or that a different provider makes a verdict reliable. We would want to see the task and the actual failures.

There is research behind taking the judge seriously as a component to evaluate. The MT-Bench and Chatbot Arena paper found that strong model judges could approximate human preferences, while documenting limitations including position, verbosity, and self-enhancement biases. That research concerns particular evaluation tasks; it does not validate the company-profile workflow in this post.

Our proposed experiment would separate two questions. First, how well does each verifier judge the same saved answers? Second, what does a complete writer–verifier workflow produce for its total cost? Changing the writer while comparing verifiers would change the material being checked and make the first question harder to answer.

Checking has its own bill.

A German-language reply described trying the pattern with two agents and finding that checking used more tokens than writing. Separately, Amrit reported difficulty managing the cost of even one agent.

We have not verified either account’s measurements. They suggest what a useful cost report should include: the first answer, every verification attempt, rejected drafts, repairs, and any paid retrieval. Human review time belongs beside that bill.

The distinction matters when a loop keeps going until the verifier approves. Approval is the loop’s stopping condition. Whether the approved answer meets the task’s requirements needs a separate check.

Parallelism also leaves work outside the model calls. One Chinese-language reply reported substantial merge, deduplication, and conflict work with ten parallel agents. That is another reason to record the human effort required to turn outputs into something usable.

A small experiment could answer more than a large dashboard.

Here is the test we would start with. This is a proposed method; we have not run it or established a winning model pairing.

1. Prepare a fixed set of distinct tasks. Use short company-source packets and one question per task. Include dates, ambiguous names, conflicting passages, and missing information. Have people check the reference answers against the saved evidence before model evaluation, and record unresolved cases.
2. Save the writers’ first answers. Ask each writer each question once under recorded settings. Keep the answer, source packet, cost, and execution status. If an answer is repaired later, retain both versions.
3. Give each verifier the same material. For every saved answer, supply the question, evidence, and a fixed rubric. Ask for supported, unsupported, or insufficient evidence, with the relevant passage. Keep human reference labels and writer identity out of the verifier’s input.
4. Report each writer–verifier pairing separately. Count unsupported answers approved out of unsupported answers judged, and supported answers rejected out of supported answers judged. Show abstentions and failed calls separately. For this test, each answer corresponds to one distinct question; repeated checks are not additional questions. Include the counts and denominators, not just percentages.
5. Test repair loops separately, with a limit. Record the maximum attempts and spending ceiling in advance. Compare the first answer with the final output using the same independent review criteria. Report total cost, elapsed time, unresolved tasks, and human review effort alongside any improvement.

For a clean comparison, the verifiers need a common set of saved answers. If one fails to judge some of them, disclose that gap and identify the shared set used for the comparison. A missing verdict is not a correct verdict or an incorrect one.

This is the kind of investigation we want to make easier with Run Any Model: ask multiple models a defined set of prompts, retain their responses and costs, and let an agent inspect that evidence later. The criteria for a correct answer still have to come from the task. Storing a model’s verdict does not turn it into an independent reference answer.

Our market-research experiment shows how we already use saved responses for further analysis. The verifier study proposed here would need its own reference set, evaluation prompts, and review before we could publish results.

If you have tried a writer–verifier setup, we would like to hear about one answer it incorrectly approved or one correct answer it rejected. The question, source passage, model settings, and resulting verdict would make a useful starting point. You can reach us at contact@runanymodel.com.