Skip to content

Notes

Relying on a vendor's AI: the evidence to ask for

In a 2026 survey, vendors supplied over half of UK financial firms' most material AI. Firms still answer for it: ask for evidence that repeats.

By Alessandro Singh and Rohith Varma, Updated , 8-minute read

In short

  • Buying the AI does not move the duty: a regulated firm answers to its customers and its supervisor for what a vendor's system does.
  • Ask for results that repeat on your own cases, tested across the whole system, with the error rate of the vendor's own checks.
  • Write it into the contract: notice and a re-test before any model change, logs you control, your own tests re-run on your schedule, who pays when it is wrong, audit rights and a way out.

Bought, not built

Much of the AI that regulated firms depend on is bought. In the Bank of England and FCA's 2026 survey of UK financial firms, third parties supplied 53% of the AI systems rated high materiality. For those, 81% of firms relied on the vendor's assurance and the contract, and 24% on deep knowledge of the system, against 77% for systems they built.Source 1

Who answers when it is wrong

The firm that uses it. In February 2024 a tribunal in British Columbia made an airline compensate a passenger misled by its website's chatbot, which said a bereavement fare could be claimed after travel. The airline argued, in effect, that the chatbot was responsible for its own actions. The tribunal held the airline responsible for all the information on its website, static page or chatbot (Moffatt v. Air Canada, 2024 BCCRT 149, a small claims decision; 812 Canadian dollars in all). The decision does not say who built the chatbot. It did not need to.Source 2

Financial regulators agree. The Prudential Regulation Authority (PRA) says boards and senior managers cannot outsource their responsibilities, and firms that outsource remain fully accountable for all their regulatory obligations (supervisory statement SS2/21). The EU's Digital Operational Resilience Act (DORA), which has applied since 17 January 2025, keeps financial firms that use outside technology services fully responsible at all times. The Mills Review of July 2026, written for the FCA's Board and published by the FCA, adds that a firm stays responsible for the outcome even when its systems run on someone else's model.Source 3Source 4Source 5

The EU AI Act also puts duties on the deployer, the firm that uses the system. For the high-risk uses it lists, such as credit scoring and life and health insurance pricing, the deployer must assign human oversight to people with the competence, training and authority to exercise it, monitor the system and keep the logs under its control for at least six months. A financial institution under EU financial services law meets the monitoring duty through the governance rules it already follows, and keeps the logs with the records those rules require. These duties apply from 2 December 2027; our note Deployed is not operational sets them out.Source 6

A vendor that sells the finished work, not the tool, answers for every result as well. In May 2025 the Solicitors Regulation Authority authorised the first AI-driven law firm in England and Wales on terms that its named solicitors are responsible for all the system's outputs, and that the system acts only when the client approves. If you buy such work as a regulated firm, the vendor's duty adds to yours; it does not replace it. So ask such a vendor how much of the work people still do, who answers when it is wrong, and what you will be shown.Source 7

Evidence that holds

A demo shows one good run. Ask for five things instead:

  • Results that repeat. Ask how often the same task gives the same result, not for the best run. In a benchmark first published in 2024, the leading AI agents of the day solved fewer than half the tasks, and the same retail task eight times running in fewer than a quarter of cases (Yao and others, ICLR, 2025). On its banking tasks, added in 2026, the best agent on the leaderboard's standard setup on 12 November 2026 solved 55% of tasks once and 35% four times running.Source 8Source 9
  • Your own cases. Public tests can flatter. The benchmark above is one: in its first version an agent that did nothing still passed 38% of the airline tasks and 6% of the retail ones, and such flaws can misstate an agent's performance by up to 100% in relative terms (Zhu and others, NeurIPS, 2025). Ask for results on your own cases, kept from the vendor until the test.Source 10
  • The error rate of the checks. Where one model marks another's answers, the marking model can be swayed: changing only the order of two answers let one model beat another on 66 of 80 questions (Wang and others, ACL, 2024). Ask how often the vendor's checks agree with your own reviewers.Source 11
  • The whole system. Test the model, its data, the workflow around it and the people who approve its work, together: as models become more capable, the Mills Review says, assurance shifts towards the whole system.Source 5
  • Someone independent. In September 2025 the UK government saw a particular role for independent assurance, but said its quality was still unclear: no certification of AI assurance skills then came from a body accredited by UKAS, the national accreditation service. Depending on the service, an assessor may need the system's limits of use, inputs and outputs, how it was built, how it is monitored and changed, and its governance documents (Department for Science, Innovation and Technology, 2025). Ask who has tested it, with what access, and whether they repeated the vendor's results.Source 12

Independent means what our thesis says: owned by no one whose AI it checks, working across models and clouds, with results others can repeat.

What goes in the contract

The contract is where evidence becomes enforceable. In the EU, DORA sets the terms for financial firms' contracts for technology that supports critical or important functions. They must include precise performance targets, notice of developments that could materially affect the service, unrestricted rights of access, inspection and audit, and an exit strategy with a transition period. The PRA expects firms to take reasonable steps to make sure material outsourcing agreements give the firm, its auditors and the regulator full access and unrestricted rights of audit, and to use those rights when appropriate.Source 4Source 3

For AI, add four terms, and take them all to your lawyer:

  • Notice before any change to the model, its settings or its guardrails, and the right to re-test before it reaches your customers.
  • Logs of every input, output and decision, kept where you control them, for as long as your rules require.
  • Your own test set, re-run on your schedule, with every result shared with you.
  • Who pays when it is wrong: the vendor's liability, its insurance and its duty to help at once in an incident.

The EU's new product liability directive covers software, AI systems included, for products placed on the market after 8 December 2026 (Directive (EU) 2024/2853, as corrected in May 2026). It compensates people, not companies; some EU countries have similar rules for companies (European Commission). For a firm's own losses, the contract matters most.Source 13Source 14

Such terms are hard to win: in the survey, negotiating them was the second most cited challenge in buying AI from third parties, counting major and moderate constraints. Where a vendor offers standard terms only, keep the controls on your side: your own test set, your own logs, a person's approval before anything reaches a customer, and a record of each gap and who accepted it.Source 1

Checks after go-live

Most firms check hardest before launch. In the same survey, more than 90% of the firms that answered used review before deployment, human approval and vendor due diligence to manage data risk in AI; about half used ongoing monitoring or periodic review. Of the 48 firms that answered on vendor updates, a quarter always or usually let high-materiality updates reach production automatically, a figure the regulators say to treat with caution.Source 1

Models drift as the world around them changes, and for higher-risk uses, in the Mills Review's words, "assurance cannot be a one-off exercise". Re-run your test set after every vendor change, and on a fixed date even when nothing has changed.Source 5

When this does not apply

Not every use needs all of this. A tool that drafts internal notes, where a person reads every word before it goes anywhere, needs a lighter check. If a wrong answer could reach a customer, a decision about a person or a regulator before anyone checks it, ask for all of it.

Questions for the vendor

Before it goes live, ask the vendor in writing:

  • What did you test, on whose cases, and how often does the same task give the same result?
  • How often do your automated checks agree with people?
  • What changes when you update the model, and how much notice will we get?
  • Who has tested the system independently of you, and can they repeat your results?
  • If it is wrong, who answers, who pays and how fast do you help?
  • If you sell the finished work, how much of it do people still do?

For investors, the list shows what a company in our thesis area, AI you can rely on, must make routine to pass two of our tests, proof in use and a route through the rules. For a company selling AI to regulated firms, it is the file the buyer will ask for.

Where to start

List the bought AI systems where a wrong answer could reach a customer, a decision about a person or a regulator. Send the vendor of the first the questions above in writing, and build a test set from your own cases before its next update.

How we help

Client resultFounder advisory, 2026

10x ARR: $200K to $2M in annual recurring revenue by November 2026 for a B2B professional-services founder. As their go-to-market team, we rebuilt pricing and qualification, provided market access, set up AI systems that gave back 20+ hours a week and closed the sales together with the founder. We also helped them raise their seed round.

The AI systems are the work of CloseWeave AI: we build AI into your systems, train your people to run it and hand it over, with an owner, limits and a log for every process.

The team has sold AI, data and cloud technology to large enterprises, governments and regulated industries.

Another of our 2026 engagements is with a company in AI-agent security, and that work was paid in equity, which one of our founders holds.

If we have an interest, we say so. If it conflicts, we step back.

More notes

All notes

Newsletter

One email a month

Our newest notes, and what we are seeing in AI, machines at work and deep technology.

We use your email only to send the monthly notes. See our privacy notice.

Sources

In the order we cite them, with the day we read each one.

  1. 1. Artificial intelligence in UK financial services 2026 (opens in a new tab)

    Bank of England and Financial Conduct Authority (a survey of 196 firms; 105 completed the full questionnaire, which most figures rest on), , Accessed , Back to the text: citation 1 of source 1, a citation 2 of source 1, b citation 3 of source 1, c

  2. 2. Moffatt v. Air Canada, 2024 BCCRT 149: reasons for decision (opens in a new tab)

    Civil Resolution Tribunal of British Columbia (a small claims decision, the tribunal's own copy), , Accessed , Back to the text, where source 2 is cited

  3. 4. Regulation (EU) 2022/2554 on digital operational resilience for the financial sector (DORA), Articles 28(1) and 30 (opens in a new tab)

    EUR-Lex, Official Journal of the European Union, , Accessed , Back to the text: citation 1 of source 4, a citation 2 of source 4, b

  4. 5. The Mills Review: AI and the future of retail financial services (opens in a new tab)

    Sheldon Mills, for the FCA Board (published by the Financial Conduct Authority), , Accessed , Back to the text: citation 1 of source 5, a citation 2 of source 5, b citation 3 of source 5, c

  5. 7. SRA approves first AI-driven law firm (opens in a new tab)

    Solicitors Regulation Authority, , Accessed , Back to the text, where source 7 is cited

  6. 8. τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (opens in a new tab)

    Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, International Conference on Learning Representations 2025 (ICLR 2025; first published as arXiv:2406.12045, 2024), , Accessed , Back to the text, where source 8 is cited

  7. 9. τ-bench leaderboard: banking tasks (opens in a new tab)

    Sierra (a company that sells AI agents), which runs the benchmark, Accessed , Back to the text, where source 9 is cited

  8. 10. Establishing Best Practices in Building Rigorous Agentic Benchmarks (opens in a new tab)

    Yuxuan Zhu, Tengjun Jin, Yada Pruksachatkun and 23 others, NeurIPS 2025, Datasets and Benchmarks Track (arXiv:2507.02825), , Accessed , Back to the text, where source 10 is cited

  9. 11. Large Language Models are not Fair Evaluators (opens in a new tab)

    Peiyi Wang, Lei Li, Liang Chen and eight others, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pages 9440 to 9450 (arXiv:2305.17926), , Accessed , Back to the text, where source 11 is cited

  10. 12. Trusted third-party AI assurance roadmap (opens in a new tab)

    Department for Science, Innovation and Technology, , Accessed , Back to the text, where source 12 is cited

Tell us what you are working on.

A founder replies within one working day.