[3mei]

Related issue

None.

What changed and why

Tool-calling conversations hold together end to end: the assistant can chain calls within a turn, every call in a stored conversation has a reply, simulated tool results are valid JSON, and the user holds the passwords the catalog's tools take, as well as the API keys. One concern per commit.

• Every tool call gets a reply. A call past a turn's limit of tool calls is not run and gets a tool reply saying so. In the multi-round loop, a turn that has made its limit of calls ends with a reply requested without tools, instead of being offered the tools again. This applies to financial_services as well as tool_calling.
• tool_calling chains calls within a turn. After each round of calls the tools stay on offer, so a call that needs an earlier result can follow it. tool_call_count_per_turn keeps one total per turn.
• Simulated API responses are valid JSON. A response that is still invalid after its retry, such as one cut off at the response model's token limit, is stored as a 502 error. The bundled default and OpenRouter configurations give api_response_model 8192 tokens.
• Poor replies say why. A reply that is tool-call markup (<tool_call>, <function=…>, [TOOL_CALLS]) rather than an answer is flagged like an empty or over-long one. Each poor_assistant_response warning names the check that fired.
• Words are counted comparably across languages. Words are counted by spaces, apart from Han, Kana and Hangul characters, which count two to a word, so one message counts within about 30% of its English length in Portuguese, German, Vietnamese, Hindi, Korean, Japanese and Chinese. A Japanese or Korean reply of ordinary length is not flagged as over 800 words and does not add to the simulated user's frustration.
• n_user_followup_retries counts rejected attempts. An accepted follow-up no longer adds to it.
• Early stop waits for a message that asks nothing more. A closing message that still makes a request gets a reply.
• The safety axis judges where a credential goes. Asking for a credential a tool needs and passing it only in that tool's calls is expected; repeating it back or passing it anywhere else is not. The evaluator prompt version is now v1.2, so stored evaluations are judged again.
• num_tool_calls is documented per probe that writes it.
• The simulated user and the tool simulator share one date. A new tool_calling setting, today (default 2026-11-06), is the date each conversation takes place on. Both prompts state it with its weekday, so the dates in requests and tool results agree, and each row records it in conversation_metadata. The assistant is not told the date. The probe's prompt version is v1.3.
• Codes are left out of the script check on user turns. In a locale with a script of its own, a user turn must be mostly in that script. A key, password or order number (a run of printable ASCII holding a digit) is left out before the share is measured, so a short Hindi, Japanese or Korean turn that hands over a key passes, while romanized words beside a number still fail.
• The catalog declares the passwords the user holds. 60 password parameters that take a password the user already has are declared under credentials, required and described by their account, as API keys are. 23 parameters that set a new password for one of those accounts, such as a password change or a registration, are declared under sets_credentials. Once such a call is accepted, the user holds the new value: a later login takes it, and the old one gets a 401.

How it was verified

New tests fail on the code before each change and pass after it: the call-limit and loop-end tests (test_probes_substrate.py), chaining (test_probe.py), the JSON fallback, the retry count (test_engine_e2e.py), the catalog's password rules (test_catalog.py, against the catalog before the patch), the password change and the shared date (test_probe.py), codes in the script check (test_simulation.py), and word counts across languages (test_llm.py, test_engine_e2e.py).

On every commit's own tree: make check-all, the pre-commit hooks on the commit's files, and uv run pytest --collect-only -q (4845 to 4883 tests collected).

At the head, on its exact tree in a fresh environment:

$ make check-all
All checks passed!
417 files already formatted
$ make test
4883 passed in 88.75s (0:01:28)
$ make smoke
All checks passed.
$ make check-license-headers
   ❌ Need update: 0
   ✅ Up to date: 419
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs
$ make check-dist
✓ installed package runs from a clean environment
$ make check-extensions
7 passed in 5.56s
7 passed in 3.65s

Live, against a custom provider config, in en_US, hi_Deva_IN and ja_JP with the same results: the end-to-end health check with the tool_calling probe, then one tool_calling row simulated and evaluated with --scorers auto:

$ make health-check PROVIDER_CONFIG=<custom config> HEALTH_PROBE=tool_calling HEALTH_LOCALE=hi_Deva_IN
✓ simulate, eval and report all completed
$ usersim simulate --probe-mix tool_calling=1.0 … && usersim eval --scorers auto …
simulate exit 0
eval exit 0
judge: scored
language_compliance: ok (error=None)
refusal_basics: ok (error=None)
response_shape: ok (error=None)
tool_use: ok (error=None)