Skip to content

Area 1 of 3

AI you can rely on

Regulated firms are putting AI into work where a wrong answer has a legal owner. That owner needs evidence from someone independent of the AI: how it was tested, what it did and why. We back the companies that make that evidence routine.

By Alessandro Singh and Rohith Varma, Last reviewed , 11-minute read

What we mean

We mean measurement, assurance and control for AI and AI agents in regulated work: the tests before a system goes live, the checks while it runs, and the record of what it did and why.

The market has several names for this work: AI assurance, AI evaluation, model risk management, AI observability and responsible AI.

By independent we mean three things: the company is not owned by anyone whose AI it checks, its tools work across models and clouds, and someone else can repeat its results.

This page is for you if one regulated firm runs your product in production and the second has not signed yet, or went live only with your engineers on site.

What we look for

Our thesis is that a technology company is proven at the second deployment. These are its five tests, as they apply here.

  • Proof in use. A regulated customer runs the product on a live system, and keeps running it after the pilot. The product tests the whole system together (model, data, workflow and human approvals), and its results repeat. A single score hides how often a system fails on the next run. In a benchmark first published in 2024, the leading AI agents of the day solved fewer than half the tasks, and solved the same retail task eight times running in fewer than a quarter of cases. On the banking tasks added in 2026, the best agent solved 55% of tasks once and 35% four times running (the leaderboard's standard setup, 12 November 2026).Source 1Source 2
  • A buyer who pays. The buyer answers for the outcome or has to sign it off: the senior manager accountable for the service, and the risk, compliance, model validation and internal audit teams who approve it. One of them holds a budget for it. A product sold only to developers competes with what they can build themselves.
  • A way to deploy again. The second regulated customer needs far less custom work than the first. The tests, the mapping to the rules and the reports carry over.
  • A way to support it. Models, data and rules change. The checks run again after every change, without the company's engineers on site. The Mills Review of AI in retail financial services, written for the FCA's Board and published by the FCA in July 2026, says "controls are part of the deployed system", and that evaluation, monitoring, audit and escalation shape whether firms can use AI safely in regulated services.Source 3
  • A route through the rules. The company's evidence has been used where it counts: in a regulated customer's approval to go live, in an audit or in front of a supervisor, and it held.

Check your own company against the five tests.

When AI does the work

A company that sells software can leave the checking to its customer. A company that sells the finished work answers for every result. In May 2025 the Solicitors Regulation Authority authorised the first AI-driven law firm in England and Wales on those terms: its solicitors answer for everything the system produces, and the system may not propose case law. As more companies are paid for the work and not for the tool, the evidence this area is about becomes part of what they sell: how the system was tested, what it did and who checked it. So we ask every such company three things: how much of the work people still do, who answers when it is wrong, and what a client is shown.Source 4

Buyers have started to pay this way. In October 2026 Sierra, which charges mainly for completed customer-service work, reported more than $400 million in annual recurring revenue, a little over two and a half years after it launched.Source 5Source 6

In September 2026 Salesforce completed its purchase of Fin, which charges from 99 cents for each outcome its AI agent delivers; the price agreed in June was about $3.6 billion.Source 7Source 8Source 9Source 10

It is still early: in ICONIQ's 2026 survey of executives at software companies building AI products, 23% reported pricing by results, up from 18% six months before. The average gross margin on AI products was 45% in 2025; the same executives projected 53% for 2026.Source 11

Sequoia Capital, which made the investor's case in March 2026, puts spending on services at six dollars for every dollar spent on software. The figure is Sequoia's own, and it gives no source for it.Source 12

Why now

Testing once is not enough. The Mills Review said that for higher-risk uses "assurance cannot be a one-off exercise", and that a useful answer is not enough if nobody can reconstruct what it rested on.Source 3

Use is wide, and recorded failures are rising. In the Bank of England and FCA's 2026 survey, 86% of the firms that responded were using AI. The AI Incident Database recorded 362 incidents in 2025, up from 233 in 2024, as reported in Stanford's 2026 AI Index.Source 13Source 14

The law is arriving at different speeds. In the EU, the AI Act requires high-risk systems to be tested before they reach the market and to keep automatic logs. Those duties apply from 2 December 2027 for the high-risk uses the Act lists, and from 2 August 2028 for AI inside regulated products. One testing duty already applies: since 2 August 2025 the makers of the largest new general-purpose models have had to evaluate them and test them as an attacker would; models already on the market have until August 2027.Source 15Source 16Source 17Source 18Source 19

In the UK, the Prudential Regulation Authority's model risk principles for banks already include independent validation of models. In the US, the bank regulators' April 2026 guidance on model risk leaves generative AI and AI agents outside its scope. The firms that need evidence first are the ones that answer for the outcome, whatever the date.Source 20Source 21

Large companies are buying the testers. In 2026 a model developer, a network company, a monitoring company and a security company each agreed to buy a firm that tests or monitors AI, one of them for $915 million. The deals show the capability is worth owning.Source 22Source 23Source 24Source 25

The evidence

Thirteen findings, in three groups. Each line is one finding, with its source.

Five findings on what firms and model developers are doing:

  • Agents are in use, within tight limits. In the Bank of England and FCA's 2026 survey, 80% of the firms that completed the full questionnaire had deployed AI agents, and 2% had agents with broad autonomy.Source 13
  • Much of the AI comes from outside. In the same survey, third parties supplied 39% of all the AI systems firms reported, and 53% of the high-materiality ones.Source 13
  • Firms lean on the vendor's word. For the high-materiality AI they bought from outside, 81% of firms relied on the vendor's assurance and the contract. For high-materiality systems they built themselves, 77% relied on their own deep knowledge; for those from third parties, 24% did. More than half (54%) named applying model risk principles to vendors' models as a challenge.Source 13
  • Checks before launch outrun checks after. In the same survey, review before deployment, human approval and vendor due diligence were each used by more than 90% of the firms that answered, to manage data risk in AI. Ongoing monitoring, leakage testing and periodic review were each used by about half.Source 13
  • Policies have spread; published test results have not. In a survey of business leaders run by the AI Index with a consultancy, the share of organisations with no responsible AI policy fell from 24% in 2024 to 11% in 2025. The developers of the leading models still rarely report results on responsible AI benchmarks (Stanford AI Index, 2026).Source 14

Six peer-reviewed papers on what measuring AI can and cannot do:

  • Software judges can match people. Strong language models, used to mark other models' answers, agreed with human preferences more than 80% of the time, the level at which people agree with each other (Zheng and others, NeurIPS, 2023).Source 26
  • And they can be swayed. Changing only the order of two answers let one model beat another on 66 of 80 questions (Wang and others, ACL, 2024).Source 27
  • Models can learn a public test. On a new test written to match a public arithmetic test, some models' accuracy fell by up to 8% (Zhang and others, NeurIPS, 2024).Source 28
  • The same settings give different answers. Five language models set to give the same answer every time varied in accuracy by up to 15% between runs (Atil and others, Eval4NLP workshop, 2025).Source 29
  • Public tests can flatter. The agent benchmark cited above is one: in its first version an agent that did nothing still passed 38% of the airline tasks and 6% of the retail ones. Flaws of that kind can misstate an agent's performance by up to 100% in relative terms (Zhu and others, NeurIPS, 2025).Source 30
  • Agent systems fail in known ways. A study that annotated more than 1,600 recorded runs from seven multi-agent frameworks set out 14 kinds of failure (Cemri and others, NeurIPS, 2025).Source 31

Two findings on the market and the law:

  • Suppliers are many, specialists few. The UK government estimated that 524 firms supplied AI assurance in 2024, 84 of them specialists. In 2025 it set out steps towards a profession of AI assurance, to raise the quality of third-party assurance.Source 32Source 33
  • Liability is moving to the maker. The EU's new product liability directive covers software, AI systems included, and applies to products placed on the market after 8 December 2026 (Directive (EU) 2024/2853, as corrected in May 2026).Source 34Source 35

Taken together: regulated firms check AI before launch far more than after it and take the vendor's word for much of what they buy, the common ways of scoring it are fragile, and liability is moving towards those who make it. What is missing is the record of what a system did, and tests that repeat.

Where this could be wrong

Rules can slip. The EU moved its high-risk duties to December 2027 and August 2028, and Colorado rewrote its AI Act weeks before it was due to take effect: the replacement, on automated decisions, starts in January 2027. So we do not back companies that need a deadline to sell.Source 17Source 36Source 37

Tools get bundled. The large platforms now ship evaluation with their models, and open-source tools are free. So we look for what a platform cannot give: independence from the AI being measured, results that repeat, and evidence a regulator accepts.Source 38Source 39Source 40

An approval may not carry over. A system cleared to go live for one use can need clearing again for the next. So we ask whether the second use needs a new approval, and who gives it.

The acquirers may be the platforms. Each of the four deals above folds a tester into a larger product, and a tester bought by a maker of AI stops being independent. So we ask who would buy the company, and whether its customers would still trust its evidence afterwards.

The measuring itself may not hold. The research above shows that software judges can be swayed and that the same settings can give different answers. So we ask each company for its own error rate: how often its checks agree with people, and how far its results move between runs.

What would change our mind: if by our review in May 2027 independent tools are still sold mainly to developers, and the people who answer for the outcome are still not buying, we are wrong about the buyer, and we will say so here.

Last reviewed We review this page every six months. When we change our view, the old view stays here, with the date and the reason.

What we bring

The team has sold AI, data and cloud technology to large enterprises, governments and regulated industries. For a company in this area, we do the commercial work in three parts. The buyers: the people who answer for the risk in banks, insurers and other regulated firms. The route in: procurement, the rules on where data is kept, and compliance. The people who run it: senior leaders, part-time until your permanent team is in place.

Client resultFounder advisory, 2026

30x pipeline. A B2B company in AI-agent security

Starting point
$1M of pipeline, with their founders doing the selling.
What we did
We became their go-to-market team and built the revenue engine with them: we helped hire three salespeople, set up the sales processes and systems, coached the team and provided market access.
The result
$30M of pipeline and $1.6M in revenue during 2026. We closed the sales together with their founders and helped them raise their Series A.

By November 2026 their own salespeople were closing without the founders.

When it is time to raise, we help you raise: the story, the numbers, the data room, an investor strategy and support through the round. Introductions and transaction mandates are arranged through regulated partners.

For funds and corporate venture teams, we scout and screen companies in this area against your thesis, and write up where each stands on the five tests. The note is factual and commercial, with no recommendation to invest or pass.

We also invest our own money when the fit is right, and can take part of a fee in equity. Size and terms are agreed case by case, in writing. Every stake is disclosed in writing to every client involved.

If we have an interest, we say so. If it conflicts, we step back.

One of our 2026 engagements is with a company in AI-agent security, and that work was paid in equity, which one of our founders holds.

The founders have closed deals with teams at

Enterprise and government deals closed by the two founders.

One regulated customer live and the second not signed yet? Tell us where you are.

Newsletter

One email a month

Our newest notes, and what we are seeing in AI, machines at work and deep technology.

We use your email only to send the monthly notes. See our privacy notice.

Sources

In the order we cite them, with the day we read each one.

  1. 1. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (opens in a new tab)

    Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, International Conference on Learning Representations 2025 (ICLR 2025; first published as arXiv:2406.12045, 2024), , Accessed , Back to the text, where source 1 is cited

  2. 2. τ-bench leaderboard: banking tasks (opens in a new tab)

    Sierra (a company that sells AI agents), which runs the benchmark, Accessed , Back to the text, where source 2 is cited

  3. 3. The Mills Review: AI and the future of retail financial services (opens in a new tab)

    Sheldon Mills, for the FCA Board (published by the Financial Conduct Authority), , Accessed , Back to the text: citation 1 of source 3, a citation 2 of source 3, b

  4. 4. SRA approves first AI-driven law firm (opens in a new tab)

    Solicitors Regulation Authority, , Accessed , Back to the text, where source 4 is cited

  5. 5. Our customers' ambition is accelerating our growth (opens in a new tab)

    Sierra (company announcement; its own unaudited figure), , Accessed , Back to the text, where source 5 is cited

  6. 6. Outcome-based pricing for AI agents (opens in a new tab)

    Sierra (company page), , Accessed , Back to the text, where source 6 is cited

  7. 7. Salesforce Completes Acquisition of Fin (opens in a new tab)

    Salesforce (press release), , Accessed , Back to the text, where source 7 is cited

  8. 8. Salesforce Signs Definitive Agreement to Acquire Fin (opens in a new tab)

    Salesforce (press release), , Accessed , Back to the text, where source 8 is cited

  9. 10. Fin AI Agent Pricing (opens in a new tab)

    Fin (company page), Accessed , Back to the text, where source 10 is cited

  10. 11. 2026 State of AI Report: The Builder's Economy (opens in a new tab)

    ICONIQ (a survey by an investor), , Accessed , Back to the text, where source 11 is cited

  11. 12. Services: The New Software (opens in a new tab)

    Sequoia Capital (an essay by an investor, Julien Bek), , Accessed , Back to the text, where source 12 is cited

  12. 13. Artificial intelligence in UK financial services 2026 (opens in a new tab)

    Bank of England and Financial Conduct Authority (a survey of 196 firms; 105 completed the full questionnaire, which most figures rest on), , Accessed , Back to the text: citation 1 of source 13, a citation 2 of source 13, b citation 3 of source 13, c citation 4 of source 13, d citation 5 of source 13, e

  13. 14. The 2026 AI Index Report: Responsible AI (opens in a new tab)

    Stanford Institute for Human-Centered Artificial Intelligence, , Accessed , Back to the text: citation 1 of source 14, a citation 2 of source 14, b

  14. 17. Regulation (EU) 2026/1744 (Digital Omnibus on AI) (opens in a new tab)

    EUR-Lex, Official Journal of the European Union, , Accessed , Back to the text: citation 1 of source 17, a citation 2 of source 17, b

  15. 20. SS1/23: Model risk management principles for banks (opens in a new tab)

    Bank of England, Prudential Regulation Authority, , Accessed , Back to the text, where source 20 is cited

  16. 21. SR 26-2: Revised Guidance on Model Risk Management (opens in a new tab)

    Board of Governors of the Federal Reserve System, , Accessed , Back to the text, where source 21 is cited

  17. 22. Promptfoo is joining OpenAI (opens in a new tab)

    Promptfoo, , Accessed , Back to the text, where source 22 is cited

  18. 26. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (opens in a new tab)

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng and ten others, NeurIPS 2023 Datasets and Benchmarks Track (arXiv:2306.05685), , Accessed , Back to the text, where source 26 is cited

  19. 27. Large Language Models are not Fair Evaluators (opens in a new tab)

    Peiyi Wang, Lei Li, Liang Chen and eight others, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 9440 to 9450 (arXiv:2305.17926), , Accessed , Back to the text, where source 27 is cited

  20. 28. A Careful Examination of Large Language Model Performance on Grade School Arithmetic (opens in a new tab)

    Hugh Zhang, Jeff Da, Dean Lee and twelve others, NeurIPS 2024 Datasets and Benchmarks Track (arXiv:2405.00332), , Accessed , Back to the text, where source 28 is cited

  21. 29. Non-Determinism of "Deterministic" LLM System Settings in Hosted Environments (opens in a new tab)

    Berk Atil and others, Proceedings of the 5th Workshop on Evaluation and Comparison of NLP Systems (Eval4NLP 2025), pages 135 to 148 (a workshop paper; arXiv:2408.04667), , Accessed , Back to the text, where source 29 is cited

  22. 30. Establishing Best Practices in Building Rigorous Agentic Benchmarks (opens in a new tab)

    Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun and 23 others, NeurIPS 2025, Datasets and Benchmarks Track (arXiv:2507.02825), , Accessed , Back to the text, where source 30 is cited

  23. 31. Why Do Multi-Agent LLM Systems Fail? (opens in a new tab)

    Mert Cemri, Melissa Z. Pan, Shuyi Yang and ten others, NeurIPS 2025, Datasets and Benchmarks Track (arXiv:2503.13657), , Accessed , Back to the text, where source 31 is cited

  24. 32. Assuring a Responsible Future for AI (opens in a new tab)

    Department for Science, Innovation and Technology (a government estimate), , Accessed , Back to the text, where source 32 is cited

  25. 33. Trusted third-party AI assurance roadmap (opens in a new tab)

    Department for Science, Innovation and Technology, , Accessed , Back to the text, where source 33 is cited

  26. 36. SB26-189: Automated Decision-Making Technology (opens in a new tab)

    Colorado General Assembly, , Accessed , Back to the text, where source 36 is cited

  27. 38. Evaluate the performance of Amazon Bedrock resources (opens in a new tab)

    Amazon Web Services, Amazon Bedrock User Guide, Accessed , Back to the text, where source 38 is cited

  28. 39. Observability in generative AI (opens in a new tab)

    Microsoft Learn, Microsoft Foundry documentation, , Accessed , Back to the text, where source 39 is cited

Tell us what you are working on.

A founder replies within one working day.