Checking a Dutch supplier before payment: web search alone, and web search plus AstraNL

A pre-registered comparison on the same 61 invoice tasks, each run once with web search alone and once with web search plus AstraNL, with the same public sources.

Setup

• Same agent, same prompt, same answer schema in both conditions: an accounts-payable assistant must apply a written four-check payment policy to one supplier invoice and answer APPROVE_PAYMENT or HOLD with the failed checks.
• Web search alone: the agent may search and read the public web, including the KVK and VIES sites. Web search plus AstraNL: the same, and the public AstraNL MCP server is attached. The prompt does not name AstraNL.
• Agent: gpt-6-astra through the Codex command line with live web search, read-only sandbox. Runs from 2026-09-30T09:14:15Z to 2026-09-30T23:21:47Z, order randomised, paced.
• Invoices are test cases constructed by AstraNL from public register data. No company named in them issued these invoices, and a mismatch in a test case says nothing about that company.

Results

A critical error is a false approval or a failing check the agent did not list. Cost per run is 0.01 EUR per wall-clock second plus 2 EUR per million tokens the command line reports; it is a fixed accounting rule, not an invoice.

What the numbers mean, and what changed during the runs

• How to read the verdict: the target is met on its second clause only. Web search alone produced no fully correct answer, so its cost per correct completion is undefined and any finite cost is lower. The first clause, 30 percent fewer critical errors, is not met: both conditions made zero critical errors and zero false approvals.
• Why web search alone has no correct answer: in all 61 runs it answered HOLD and listed checks it could not verify as failed, three or four of the four checks in 60 runs, because it could not get a usable answer from the register or from VIES and the policy counts an unverified check as failed. That is safe, but it blocks every payment, including the 11 invoices from suppliers that pass all four checks.
• What the tool changed: with the AstraNL server attached, the same agent called the tool in 60 of 61 runs and gave the expected decision with the expected failed checks in 60 of 61. The median run took 27 seconds against 44.
• The runs were paused and restarted three times before any scoring, each time with the partial results archived and never scored: at 08:30 UTC because version 1 of the tool passed on register data in a way the KVK terms of use do not allow; at 08:50 UTC because a VIES congestion error was read as an invalid VAT number; at 09:01 UTC because an independent review found normalisation and completeness defects. The scored runs started at 09:14 UTC on tool version 3 with the same tasks, prompt, conditions, scorer and target.
• At 21:41 UTC, while the runs were going, the tool was extended with a check of the Dutch insolvency register, tool version 4. Five AstraNL runs used version 4 and all five were correct. The payment policy in the tasks does not cover insolvency.
• The agent reported a second connector, labelled AstraNL, that the ChatGPT account used by the command line makes available in both conditions. Every call to it failed on the approval policy: all 61 web-search-alone runs show one failed tool call and no AstraNL data. In one AstraNL run the agent chose that blocked connector instead of the attached server, answered without register data, and is counted as incorrect.
• Task CPB1-010: the web-search-alone run timed out on its first attempt. The second attempt that the pre-declared rules allow ran at 23:21 UTC, after the other runs, and answered; its cost includes the time of the failed attempt. An earlier version of this page, published at about 23:07 UTC, scored before that second attempt and left the task out of both conditions.

Correct decisions by task category

Limits

• One agent model and one command-line client. Other models may do better or worse in either condition.
• Test invoices, not live payments. The policy covers register status, name and VAT only; bank account ownership and solvency are outside it.
• AstraNL built both the tasks and the tool. The runner, the scorer and the task generator are published below with the sha256 of every file, including the task and answer files. The task file itself holds Dutch register data and is not republished, in line with the KVK terms of use; anyone with KVK API access can build a comparable set with the published generator.

Reproduce

• tasks.json sha256 4f077efa8ff7f0ae54647ed501abc725d29d1bdf7c50e0d073a2bad1698dbdb2, not published
• truth.json sha256 acb4814a87b6729fbcf5d8aed5086b3537de1ab90e44049a377621b53c439f9e, not published
• runner.py sha256 09a438c54d6f94f509ee5893c0bc666c98c0cb3536eb1529510dd2c79bd788c1
• score.py sha256 c76bdd563b96bb302223aa916a2b673b720ab0fa830dfd3aaf3c9802f023fd15
• generate.py sha256 71a6cb7cdd3f95db2de840078ce7329a5ba30126e689150942f79ebf7c4d2af5
• build_pool.py sha256 79eef1a4815818d40bbc3f9e3d2a32a9a1152bceb2b17915b114c78eb46cd922

The tool itself: check a Dutch supplier before you pay.

Rendered 2026-09-30 23:22 UTC from the scorer output; decision 430.