We used Run Any Model to research its own market
We built a tool to ask multiple AI models and keep their answers. Then we used it to research the market we wanted to enter.
Before we shared Run Any Model, we used it to look for tools like Run Any Model.
We already had a working tool: we could send questions to multiple AI models, keep the answers, and extract information from them later. Now we had a question of our own. What else existed? What problems did those products promise to solve? Where might ours fit?
So we put it to work on its own market. We collected 457 answers, then turned those answers into a list of products, their stated pain points, and a first pass at how they related to our idea.
Along the way, we found something we hadn’t expected: for every model we tested, the tools it recommended with Exa were almost completely different from the tools it recommended with Brave.
This is our first post about Run Any Model. The research is how we want to introduce it: by showing what we used it for, what came back, and what we could do with the answers once we had them.
A tool we could put to work
Part of the reason we built Run Any Model is that one assistant gives you one view of a question. You can rewrite your prompt or try another model, but then you have more conversations to collect and make sense of.
We wanted a way to run the combinations we chose and keep what came back. The answers would remain available for another reading, another extraction, or a different question about the same material.
For our market research, we wrote 18 questions across six needs: running prompts and saving structured results, comparing answers, combining them, finding fresh ideas, researching across sources, and avoiding the blind spots of using the same AI as everyone else.
We asked 11 models those questions with Exa and with Brave. Four models also used their provider’s built-in search. That gave us 26 combinations, with each question asked once per combination.
We ran the experiment from Claude Code through Run Any Model’s MCP connector.
At that point, we had hundreds of saved answers. To understand the market, we needed to turn them into information we could work with.
From answers to a picture of the market
A separate model, GPT-5.6 Luna, extracted product names from 456 answers. We then cleaned up duplicate names. “Opper Roundtable” and “Opper Roundtable API” counted as one product. Separate products from the same company stayed separate, and two unrelated GitHub projects called “LLM Council” needed separating.
That left 416 entries, down from 566 name groups in the original extraction. The list included tools, libraries, and workflows.
We had also asked a follow-up after every answer: according to each platform’s website, what are the three main pain points it says it solves?
That gave us a second way to look at the suggestions: through the problems they described.
| Product | Pain points described in cited follow-ups |
|---|---|
| Synero | One model’s blind spots; copying a question across models; not seeing where answers disagree |
| Kioku | Scattered AI chats; saving them manually; finding useful past answers |
| Promptfoo | Trial-and-error prompt testing; manual model comparisons; security holes in LLM apps |
| Elicit | Finding relevant papers; extracting information from them; slow literature reviews |
These are summaries of vendor claims reported by the models. They helped us organize what to investigate; we hadn’t independently verified the capabilities.
The important part of this workflow was that we could keep working with the answers we’d already collected. Changing our alias rules and recalculating the tables required no new model calls. We could revisit the answers without asking the original research questions again.
The full list contains all 416 entries and the pain points extracted from the available follow-ups. It may still contain mistaken names or links.
The surprise in the shortlists
Once we had the product names, we could look at which combinations had found the same tools.
Here is one of the questions we sent, with two saved answers that illustrate how different the recommendations could look:
The exact prompt
How can I explore a wider range of ideas by asking different LLMs, then compare what each contributes? Recommend up to six concrete tools or workflows. Explain how each fits and one important limitation. Use official product websites or documentation to support claims about each platform's capabilities, limitations and the problems it says it solves; include the relevant source URLs. Prioritize investigating your selected platforms over filling six slots. Say when a capability or claim cannot be verified. Do not treat a vendor's claim as proof that its product delivers it.
GPT-6 Sol + Exa
Its shortlist
Synero · ConvergePanel · AIPuna · LLMs Assemble · Concilium
An excerpt about Synero
Ask GPT, Claude, Gemini, and Grok the same question. It presents a consensus answer, identifies agreements and disagreements, and keeps each full answer available—useful for finding ideas that only one model raised.
GLM 5.3 + Brave
Its shortlist
LMArena · Poe · ChatHub · OpenRouter · Kagi Assistant · A manual prompt matrix
An excerpt about LMArena
Ask one prompt and get answers from two models simultaneously; its Battle mode shows two anonymous models so your vote isn't biased by brand.
These are two saved answers to the same prompt, with different models as well as different search setups. They illustrate the recommendations, not an isolated engine effect. The same-model comparisons and full answers are linked at the end of this post. Product claims in these excerpts have not been independently checked.
Starting with either answer would give you a different set of tools to investigate. To see whether that difference extended beyond these examples, we compared the product lists across the experiment.
Keeping the model the same and switching from Exa to Brave gave us almost entirely different lists of tools. We made that comparison for all 11 models. The overlap between each model’s two lists ranged from 0% to 9.3%.
For GPT-6 Sol and Grok 4.6, the difference was complete: each answered all 18 questions with both setups, and neither model recommended a single tool in common between its Exa and Brave lists.
Overlap tells us how much two lists have in common. Here’s an illustrative example, separate from our research results:
| Step | Example |
|---|---|
| Tools in the first list | A, B, C, D, E, F |
| Tools in the second list | A, B, G, H, I, J |
| Tools both lists recommend | A and B — 2 tools |
| Different tools across the two lists, counting each once | A through J — 10 tools |
| Overlap | 2 shared tools out of 10 = 20% |
A score of 0% means the lists share no tools. 100% means they contain exactly the same tools.
The table below summarizes the comparisons. We calculated an overlap score for each pair of lists, then averaged those scores within each row. A higher percentage means the lists were more similar on average.
| Comparison | Pairs of lists compared | Average overlap |
|---|---|---|
| Different models, both using Exa | 55 | 62.6% |
| Different models, both using Brave | 55 | 24.4% |
| Different models, Brave vs built-in | 40 | 23.4% |
| Same model, Exa vs Brave | 11 | 2.6% |
For each pair, we included only questions with usable extracted results on both sides. Missing results were left out, rather than treated as lists with no tools. The detailed comparisons show which questions each pair includes and how many tools the two lists share.
All 11 models on Exa named Synero, Concilium, SourceFrame, and InfraNodus at least once. None of the available Brave answers named them. Brave and built-in search had more names in common, including Promptfoo, Elicit, Poe, and Connected Papers.
Across the full list, 199 of the 416 entries appeared in just one of the 26 combinations. We don’t know how many of those less common suggestions would be useful, but they gave us leads we could have missed by starting with only one combination.
For our next round of research, we’d start with two search setups, look at what each adds, then investigate the unfamiliar suggestions ourselves.
There is a limit to what this comparison tells us. Exa supplied search results inside the model request through OpenRouter’s web plugin. With Brave, the model used our tools to search and open results. Built-in search was handled by each model’s provider. So we changed the integrations as well as the search engines. We measured the recommendations those setups produced, not which setup found the best tools.
Different questions led us to different kinds of tools
The search setup wasn’t the only thing shaping the suggestions. We had asked about six related needs, from collecting model output to finding perspectives we might otherwise miss.
| Category | What we wanted to find | Example question |
|---|---|---|
| 01Run prompts → extract → retain | Find platforms that run prompt banks across models, extract structured records and retain the evidence. | What tools let me run a set of prompts across multiple AI models, extract structured information from their responses, and save both for later analysis? |
| 02Combine answers without losing differences | Find ways to synthesize several models' responses without erasing disagreement. | How can I combine responses from several AI models into a useful summary while keeping track of disagreements? |
| 03Build a reusable research library | Find ways to turn saved AI research into a reusable, traceable dataset. | Which tools help turn a collection of AI-generated research responses into a searchable dataset linked to the original material? |
| 04Find fresh ideas across models | Find ways to gather and compare fresh ideas across models. | What tools or workflows can help me gather fresh ideas from several AI models instead of relying on one assistant? |
| 05Discover overlooked information | Find research tools and methods that surface overlooked sources and perspectives. | What tools or workflows help me research a niche topic across many sources and find useful material a single search might miss? |
| 06Avoid everyone's blind spots | Find tools and methods for making decisions from broader evidence and perspectives when widespread use of the same AI may narrow what people consider. | How can I avoid an algorithmic monoculture in my research and decisions if everyone is using the same AI assistant? What tools or workflows can help me draw on broader knowledge and independent perspectives? |
These are the opening questions as asked. Each also included the same instructions to recommend tools or workflows and explain their fit and limitations.
To make the category differences easier to see, we held the model constant in the next table: Claude Opus 5.5, which had usable results for all three questions in every category across all three search setups. Each cell shows the entries that appeared most often in that category, with ties included.
| Category | Exa | Brave Search | Built-in search |
|---|---|---|---|
| 01Run prompts → extract → retain |
|
|
|
| 02Combine answers without losing differences |
|
|
|
| 03Build a reusable research library |
|
|
|
| 04Find fresh ideas across models |
|
|
|
| 05Discover overlooked information |
|
|
|
| 06Avoid everyone's blind spots |
|
|
|
Counts refer to questions whose saved answers named the entry. They describe these recommendations, rather than product quality. All three questions were successfully read in each cell shown here.
With Brave, all three questions about running prompts, extracting information, and retaining the results named Promptfoo, Langfuse, Instructor, and llm CLI. In the fresh-ideas category, Arena appeared in all three answers. For overlooked information, Connected Papers, Elicit, and OpenAlex each appeared in two of the three.
Those suggestions gave us different paths to investigate: a workflow assembled from several tools, a place to compare models, or tools for exploring research sources. Asking about the whole workflow and asking about the underlying need brought different neighbors into view.
The search differences persisted within categories too. In fresh ideas across models, ten of the eleven same-model Exa–Brave comparisons had no tools in common. For combining answers without losing differences, six of eleven had no tools in common. Each comparison used only that category’s questions successfully read on both sides.
That gave us another lesson for our own research: vary the question as well as the search setup. A tool useful for one part of the job might never appear when we describe the whole thing at once.
What that told us about our own product
Finding hundreds of names gave us a starting point. We still wanted to understand which tools addressed similar needs and how their approaches differed.
We gave the original 566 name groups to six Claude Opus 5.5 agents, along with the same description of our idea. They classified each entry, scored its similarity to us from 0 to 5, and explained the differences. They didn’t browse the websites, so this was a first pass based on limited information.
After merging aliases, the classifications looked like this:
| Relation to Run Any Model | Deduplicated entries |
|---|---|
| Another job: literature search, notes, citation managers… | 187 |
| Related infrastructure: evals, tracing, gateways, agent frameworks | 132 |
| An application of running and comparing models | 83 |
| Similar underlying job: ask models and keep their output | 7 |
| Aliases received conflicting classifications | 6 |
| Split into a separate product; no separate classification | 1 |
Some aliases received different classifications, so those groups remain marked as mixed. For 121 of the 416 entries, at least one judgment was marked uncertain.
Consensus tools and multi-model chats came up often. They help people combine answers or talk to several models. Our own research made the role we wanted Run Any Model to play more concrete: run the combinations you choose, preserve the answers, and make them available for whatever you want to do next.
In our case, that next step was extracting names, reading pain points, and comparing suggestions. Someone else’s next step could be different.
Making it available to others
We decided to make Run Any Model a product others could use. This research is one example of the workflow we’re opening up: bring a question, run it across the models and search setups you choose, and keep the answers so you can continue working with them.
We used it to explore our own market. The result was a collection of leads, a clearer view of neighboring products, and a surprise that changed how we’d approach the next search.
This was one topic, with one sample per question. Different wording, another topic, or a later run could produce different lists. We still have to investigate the suggestions ourselves. The questions, comparisons, alias decisions, and saved answers accompany the post; you can explore them below.
The search tools we used
Exa and Brave Search are services for finding information on the web. In this experiment, they supplied the web information the AI models used to answer our questions.
- Exa: We used it through OpenRouter’s web plugin, which supplied search results inside the model request.
- Brave Search: We connected it through our browsing tools, so a model could search, open a result, and decide what to look at next.
- Built-in search: The model’s own provider handled the search, rather than using either of those services through our setup.
These are different ways to give a model access to web information. The model still produces the answer, and the search setup influences what information it has to work with.
Explore the research
- Browse all 416 entries, including their extracted pain points.
- See the full analysis, with the questions and comparison details.
- Compare the same model across search setups.
- Read the saved example answers.
- See how we merged product aliases.
- Download the analysis data.
If you have a question of your own, start an experiment in Run Any Model.