Binocs vs GPT and Claude: Caliper Lab's Independent Benchmark

An independent evaluation by Caliper Lab found that Binocs examined roughly ten times more sources and three to eight times more of the market than leading frontier AI models on the same diligence research. This piece walks through what was tested, where Binocs led, where it tied, and what that means for anyone using AI in commercial due diligence.

The question deal teams should be asking

Most of the public conversation about AI in finance is a horse race between models. Is GPT ahead this quarter? Has Claude caught up? Which one writes the cleaner memo? For a deal team, that question matters less than it looks. The model leaderboard changes every few months. What does not change is when a partner points at one number on a slide and asks where it came from. That moment is the real test of any research tool. A well-written market summary built on a handful of sources reads beautifully until confirmatory diligence, a lender, or an LP asks it to hold up. What matters is how much of the market the system actually looked at, whether the figures can be traced, and whether they can go straight into a model.

Why an independent evaluation

Every AI vendor has a benchmark it wins. Buyers have no reliable way to tell which of those numbers mean anything, because in almost every case the vendor designed the test, ran it, and graded it. Enterprise software solved this problem years ago with independent analysts. AI applications have not had an equivalent.

Caliper Lab is built to fill that gap. It is an independent AI capability measurement firm that evaluates AI products on real professional workflows rather than academic benchmarks, publishes regardless of commercial relationships, and applies the same standard to every system it tests.

Caliper Lab independently evaluated Binocs against three frontier systems: Claude Opus 4.8, GPT-5.5, and GPT Deep Research. We did not run the test and we did not grade it.

Methodology: Each system produced six research reports. Caliper's pipeline then extracted every quantified claim from those outputs, 15,383 values in total, and tagged each one with the metric it measures, the entity it describes, its unit, its time period, its source, and whether it was stated firmly or hedged. Over 5,000 citations were probed to check whether the links actually resolve. The same automated pipeline ran across all four systems.

One design choice is worth calling out. Binocs produces structured slide output, while the other three produce prose, and tables naturally state firm numbers. Caliper ran the key commitment comparison on prose-extracted values only.

Finding 1: Coverage

Binocs cited 7,722 URLs across 833 distinct source domains, with 84% of those links still reachable when probed. The frontier models drew on between 64 and 96 domains. That is roughly a tenfold difference in how much of the available evidence each system consulted before reaching a conclusion.

Think about what a standalone model is doing when it answers a diligence prompt. It finds a handful of sources, often the most prominent ones, and builds the whole narrative around them. The result can read as authoritative while resting on what amounts to two or three analyst reports.

Volume alone would not settle this, though. A system can cite a thousand pages and still have its conclusion resting on two of them. So more imprtant here is concentration. Caliper measured how evenly each system's citations were spread using a Herfindahl index(HHI), the same measure used to check whether a market is dominated by one player. Zero means perfectly spread; one means everything traces back to a single source.

Binocs scored 0.012, and it held that level across more than 7,700 citations. The frontier models, working from a fraction of the sources, were two to three times more concentrated.

In diligence terms, this is about single points of failure. A conclusion built on a few sources falls apart the moment one of them is wrong, outdated, or challenged. A conclusion spread across hundreds of independent sources has no single report holding up the whole structure, and it leaves a trail an associate can follow when someone asks.

Finding 2: Depth, how much of the market was mapped

Binocs examined 1,062 distinct entities across its reports: competitors, customers, suppliers, products, and adjacent players. GPT Deep Research covered 344, GPT-5.5 covered 202, and Claude Opus 4.8 covered 131.

The distribution matters as much as the count. Caliper found that the frontier models tended to fixate on one market leader. Claude placed 16% of all its quantified analysis on a single company. Binocs most-covered entity accounted for 6% of its values, which produces a far more balanced picture of the competitive field.

Anyone who has worked on commercial diligence understands why this has consequences. Markets are rarely shaped by one company. They are shaped by challengers, customers, suppliers, substitutes, regulators, and technologies that are only starting to matter. When analysis concentrates on the obvious incumbent, the blind spots form exactly where the interesting findings tend to sit: a smaller competitor taking share, a niche segment growing faster than the headline market, or a shift in the supply chain.

The risk in diligence is that the team reaches a confident answer from an incomplete view of the market. Before judging any AI tool's conclusion, it is worth asking how much of the landscape it actually explored.

Finding 3: Figures a deal team can actually use

The difference between useful and unusable research output is often mundane. Can the number go straight into the model, or does someone have to work out what it means first?

Caliper looked at this in two ways. The first was commitment: whether a system states a firm figure or hedges with a range or an approximation. On the prose-only comparison, Binocs stated firm point estimates 82% of the time. GPT Deep Research followed at 72%, GPT-5.5 at 59%, and Claude Opus 4.8 at 34%, with the remaining two thirds of Claude's figures given as ranges or approximations.

A point estimate such as a market size of $4.2bn in 2024 can be dropped into a model. A range of $3bn to $5bn forces the analyst to pick an assumption. An approximation reduces accountability for the figure altogether. Because this comparison excluded Binocs tables, the gap reflects how the system commits to numbers, not the format it writes in.

Binocs used 282 distinct unit types. That range is a sign of sector-specific analysis. In the satellite market Caliper used for the evaluation, that meant measuring orbital altitude, image resolution, revisit times, constellation sizes, and detection accuracy alongside the usual revenue, growth, and share figures. Standard financial vocabulary covers a summary. diligence needs the operating metrics of the industry being underwritten.

What this means for deal teams

Whichever tool your team uses, the Caliper framework gives you a practical set of questions to ask before trusting AI-generated research in a live process.

1. How many distinct sources did it draw on, and is any single one carrying the conclusion?

2. How much of the market did it map beyond the obvious leader?

3. Can each material figure be traced to a source that still resolves?

4. Are the numbers stated firmly, with units and periods, so they can go into a model as they stand?

5. Does the report agree with itself when the same figure appears twice?

These are the properties that decide whether research survives Investment Committee, confirmatory diligence, and the questions that follow a close. On the first, second, and fourth, the independent evidence puts Binocs clearly ahead of frontier models answering the same prompt. On the fifth, it matches the strongest of them.

Binocs is built to produce the verified factual baseline an investment thesis stands on. The thesis itself remains the deal team's judgement, and it should. Our job is to make sure the foundation under that judgement holds when someone tries to break it.

FAQs

Where can I get rapid commercial due diligence support for a PE deal?
The options are traditional consulting firms (Big 4 and strategy boutiques), independent expert networks, in-house deal teams, and AI-driven diligence platforms. Consultancies bring judgment and primary research but typically need several weeks and a significant fee. AI platforms like Binocs produce a sourced CDD report in 2 to 4 days at a fraction of the cost, which suits early screening and tight exclusivity windows. Many firms now use both together.

How long does commercial due diligence usually take?
A traditional CDD engagement typically runs three to six weeks depending on scope, sector complexity, and how much primary research (customer interviews, surveys) is involved. Faster turnarounds are possible with a narrower scope. AI-assisted diligence can compress the desk research portion to a few days, which leaves more time for the expert calls and judgment that still need people.

Can AI replace a consulting firm for commercial due diligence?
Not entirely, and it’s worth being clear about where the line sits. AI handles the research-heavy work well: market sizing, competitor mapping, customer and pricing analysis, and source gathering at a breadth a human team can’t match in the same time. Primary research, management access, and final investment judgment still rely on people. Many Big 4 and boutique CDD teams now use AI platforms to augment their work rather than replace it.

What should I look for in a fast CDD provider?
Speed only matters if the output holds up at investment committee. Look for source traceability (every claim linked back to where it came from), a consistent methodology across deals, clear treatment of uncertainty, and security credentials such as SOC 2 Type 2 for handling confidential deal data. Ask to see a sample report on a company you already know well and check it against your own view.

How much does commercial due diligence cost?
Traditional CDD from a consulting firm commonly ranges from tens of thousands to several hundred thousand dollars depending on deal size and scope. AI-driven platforms cost substantially less. Binocs, for example, delivers CDD at roughly 80 to 90% lower cost than traditional consulting, which makes it practical to run diligence earlier in a process or across more deals in a pipeline.