Does the language of a spatial instruction interface affect an LLM agent's ability to act?
We tested whether the language of an LLM agent's operating interface, not just its goal, changes how well it completes a multi-step spatial task. Translating the ongoing interface hurt performance far more than translating the one-time goal.
↳ 53.7% vs 31.0% success — English vs Russian full interface, matched n=300
LLM agents are often tested as though the language around a task were a transparent wrapper. That assumption becomes fragile once a model has to act over many steps: in a tool-use environment, the model keeps reading the interface while it acts, so surface text can become part of the policy rather than a neutral presentation.
We test this by rendering the same spatial game through different language surfaces while keeping the hidden world and simulator fixed. Correctness is measured from the final state, not from the model’s own explanations. In the current runs, changing the operating interface changes task success and also changes the shape of failed trajectories — including remaining distance to the target and tokens spent per successful episode. The result is not a claim that one language is inherently better. It is evidence that interface language should be treated as a controlled variable in benchmarks for spatial agents.
Research questions
RQ1. Given the same spatial task, how does changing the instruction language change an LLM’s spatial performance, error type, error severity, and token/cost efficiency?
RQ2. Is the observed effect better explained by the language of the target goal, or by the language of the repeated operating interface — tool descriptions, rules, observations, and feedback?
Hypothesis. In agentic spatial tasks, the language used for the step-by-step interface should matter more than a one-time goal translation. The agent must keep parsing action rules, observations, and simulator feedback, so small changes in the operating surface can accumulate over a plan.
The benchmark
KeyBoxWorld is a small deterministic spatial tool-use game. The model moves through rooms, picks up items, opens locked boxes when it has the right key, and places a target item into a target container. Each model response must be one JSON tool call; the simulator applies valid calls and rejects invalid ones, and the next observation is returned in the language assigned to that condition.
| World state | Rooms form a small graph. Objects are located in rooms or containers. The agent has a current room and an inventory. |
|---|---|
| Goal | The target predicate is simple: a named item must end inside a named box. The item IDs and box IDs stay symbolic across languages. |
| Action loop | At each step the model sees the goal, tool docs, recent history, and current observation, then emits one JSON call such as {"tool":"T2","args":{"item":"ITEM_41"}}. |
| Score | The scorer uses simulator state, not prose. Success means the target predicate is true; severity is measured by remaining distance to the target. |
The agent loop
- Render the prompt. The same hidden state is shown through the selected goal and interface languages: goal + tools + history + observation.
- Model chooses one action. The model must answer with a single JSON tool call, e.g. {"tool":"T1","args":{"to_room":"ROOM_B"}}.
- Simulator checks it. Valid calls change the world; invalid calls become feedback in the interface language.
- Score final state. The episode succeeds only when the target item ends inside the target box.
Experimental protocol
The benchmark is built to avoid asking the model to judge itself. The model only acts; the simulator and scorer decide what happened. This matters because language effects can appear as valid-but-wrong action choices, repeated loops, precondition failures, or extra tokens spent on unsuccessful trajectories.
| Component | Choice | Reason |
|---|---|---|
| Task generator | Deterministic KeyBoxWorld worlds | The same hidden task can be rendered through multiple languages |
| Action format | One JSON tool call per step | Validity and precondition failures are machine-checkable |
| Language manipulation | Goal language and interface language varied separately | Separates translation of the target from control-loop language |
| Scoring | Simulator final state, remaining distance, failure type, token usage | Model explanations do not determine correctness |
| Model access | OpenRouter chat-completion calls, temperature 0 | Black-box prompts, outputs, usage, and latency are enough for v1 evidence |
Example episode
A representative generated task looks like this. The object identifiers stay the same across all languages, so translations do not rename the hidden state.
| Task ID | keybox_0001 · medium difficulty |
|---|---|
| Goal | ITEM_49 inside BOX_49 |
| Initial room | ROOM_A |
| Rooms | ROOM_A, ROOM_B, ROOM_C, ROOM_D |
| Observed run | Russian goal, English interface solved in 10 model steps. |
Language surfaces
The multilingual version keeps the same game and changes language through condition names. For any non-English language L, L_all means the goal, rules, tools, observations, and feedback are all in that language. L_goal_en_tools keeps only the goal in L while the operating interface stays English. en_goal_L_tools keeps the goal English while the operating interface changes to L.
| Language | Goal sentence | Move tool | Response instruction |
|---|---|---|---|
| English | Put ITEM_41 into BOX_41. | T1(to_room): Move to an adjacent room. | Reply with one JSON tool call. |
| Russian | Положи ITEM_41 в BOX_41. | T1(to_room): Перейти в соседнюю комнату. | Ответь одним JSON-вызовом инструмента. |
| Spanish | Pon ITEM_41 dentro de BOX_41. | T1(to_room): Muévete a una habitación adyacente. | Responde con una sola llamada de herramienta en JSON. |
| German | Lege ITEM_41 in BOX_41. | T1(to_room): Gehe in einen benachbarten Raum. | Antworte mit genau einem JSON-Werkzeugaufruf. |
| French | Mets ITEM_41 dans BOX_41. | T1(to_room): Va dans une pièce adjacente. | Réponds avec un seul appel d'outil en JSON. |
| Armenian | ITEM_41 տեղադրիր BOX_41 տարայի մեջ. | T1(to_room): Տեղափոխվիր հարակից սենյակ. | Պատասխանիր մեկ JSON գործիքի կանչով. |
Results
Complete matched-language anchor
The complete KeyBoxWorld run has 300 hidden tasks under four language conditions, for 1,200 completed scored episodes. This is the cleanest within-language comparison because every condition covers the full task set.
The mixed conditions are the key diagnostic. A Russian goal with English tools remains close to the English control. An English goal with Russian tools falls near the full Russian interface. This pattern points away from a simple translation-only explanation and toward the operating interface as a major source of the effect.
In aggregate, the English full-interface condition solved 53.7% of episodes. The full Russian interface solved 31.0%. When only the goal was Russian and the operating interface stayed English, success was 50.7%; when the goal was English but the operating interface was Russian, success was 32.7%. The asymmetry is the central result of the anchor run.
| Condition | Episodes | Success rate | Avg. final distance | Tokens per success |
|---|---|---|---|---|
| English full interface | 300 | 53.7% | 2.510 | 8,707.7 |
| Russian goal, English interface | 300 | 50.7% | 2.623 | 9,126.5 |
| English goal, Russian interface | 300 | 32.7% | 3.327 | 18,011.3 |
| Russian full interface | 300 | 31.0% | 3.428 | 19,282.3 |
| Matched comparison | Matched tasks | Left only | Right only | Both solved | Both failed | Exact p |
|---|---|---|---|---|---|---|
| English full vs. Russian full | 300 | 92 | 24 | 69 | 115 | <0.1% |
| Russian full vs. Russian goal + English interface | 300 | 22 | 81 | 71 | 126 | <0.1% |
| English full vs. English goal + Russian interface | 300 | 87 | 24 | 74 | 115 | <0.1% |
Anchor diagnostics
The English/Russian run is useful because it includes difficulty and failure-type views across the full task set. Difficulty shows whether the gap only appears in harder plans. Failure types show whether the model is failing the interface contract, getting stuck in loops, or ending in the wrong world state. Across all four conditions, easy tasks (45 episodes each) solved at 100%; the gap opens up at medium difficulty (210 episodes each: 53.3% English full vs. 20–21.4% for the two Russian-interface conditions vs. 49.0% for Russian goal + English interface) and narrows again at hard difficulty, where every condition scores in the 8.9–17.8% range.
By failure type, the English full-interface condition’s non-successes are mostly precondition_failed (29.7%) and loop (12.7%). Both Russian-interface conditions shift heavily toward loop (25.7% for English goal + Russian interface, 28.0% for Russian full) alongside a similar precondition_failed share, meaning the Russian-interface agent is more often getting stuck repeating actions rather than just picking the wrong valid action.
Multilingual extension
The multilingual extension applies the same condition design beyond the original two-language anchor, across English, Russian, Spanish, German, French, and Armenian, with a harder task mix. It is designed as the current main multilingual measurement path. The present scored artifact is still partial because OpenRouter returned a key-total-limit 403 before the full 3,840-episode matrix completed — clean scored episodes: 1,146. Because task coverage can differ by run, matched-pair counts are used where possible.
In the current artifact, Armenian is included as a full Armenian interface, an Armenian-goal/English-interface condition, and an English-goal/Armenian-interface condition — the same three-surface design used for the other four languages.
| Language | Condition surfaces present | Clean episodes | Included surfaces |
|---|---|---|---|
| English | 1 | 72 | Full English interface |
| Russian | 3 | 214 | Russian full interface; Russian goal + English interface; English goal + Russian interface |
| Spanish | 3 | 216 | Spanish full interface; Spanish goal + English interface; English goal + Spanish interface |
| German | 3 | 216 | German full interface; German goal + English interface; English goal + German interface |
| French | 3 | 216 | French full interface; French goal + English interface; English goal + French interface |
| Armenian | 3 | 212 | Armenian full interface; Armenian goal + English interface; English goal + Armenian interface |
Goal language versus interface language
The point of the multilingual matrix is not to rank languages, but to test whether translated goals and translated operating interfaces move performance differently on the same hidden tasks, across two model families: qwen-27b and gpt-oss-20b.
| Model | Language | Full interface | Foreign goal + English interface | English goal + foreign interface | Drop from English: goal only | Drop from English: interface |
|---|---|---|---|---|---|---|
| gpt-oss-20b | Russian | 5.7% | 2.9% | 0.0% | -0.1% | 2.8% |
| gpt-oss-20b | Spanish | 0.0% | 2.8% | 5.6% | 0.0% | -2.8% |
| gpt-oss-20b | German | 2.8% | 0.0% | 0.0% | 2.8% | 2.8% |
| gpt-oss-20b | French | 2.8% | 5.6% | 0.0% | -2.8% | 2.8% |
| gpt-oss-20b | Armenian | 8.6% | 2.9% | 5.6% | -0.1% | -2.8% |
| qwen-27b | Russian | 11.1% | 30.6% | 27.8% | 8.3% | 11.1% |
| qwen-27b | Spanish | 22.2% | 30.6% | 13.9% | 8.3% | 25.0% |
| qwen-27b | German | 16.7% | 33.3% | 13.9% | 5.6% | 25.0% |
| qwen-27b | French | 30.6% | 36.1% | 27.8% | 2.8% | 11.1% |
| qwen-27b | Armenian | 17.1% | 14.3% | 11.1% | 24.6% | 27.8% |
Both “drop from English” columns are approximate success-rate loss relative to each model’s English full-interface control. For qwen-27b, the interface-language drop is consistently larger than the goal-language drop across all five languages (11.1–27.8% vs. 2.8–24.6%); for gpt-oss-20b, both model and non-English full-interface success rates are low enough (0–8.6%) that the pattern is noisier, but points the same direction for Russian, German, and French.
Multilingual diagnostics
The multilingual matrix is incomplete, so these diagnostics are not treated as final estimates. They are still useful for deciding what to run next: English-interface mixed conditions are competitive across several goal languages, while full non-English interfaces and non-English operating interfaces often show lower success and higher remaining distance. For qwen-27b, every language’s easy-difficulty tasks (3 episodes each) solved at 100% regardless of condition, and medium-difficulty success ranged from 13.3% (Armenian goal, English interface) up to 73.3% (English full interface); hard-tier tasks solved only for Spanish goal + English interface (11.1%), with every other condition at 0%. For gpt-oss-20b, medium-difficulty success topped out at 6.7% (English and French goal + English interface), with most conditions at 0% across both medium and hard tiers.
By failure type, qwen-27b’s full-interface conditions succeed at 11.1–38.9% (English highest, Spanish lowest at 22.2%), with precondition_failed as the leading non-success cause in every language (38.9–66.7%). gpt-oss-20b’s full-interface conditions succeed far less often (2.8–8.6%), and invalid_json becomes a leading failure mode for English, French, German, and Russian (36.1–47.2% of episodes) — a failure type that barely appears for qwen-27b.
Matched multilingual comparisons
Because the multilingual matrix is incomplete, matched comparisons are safer than reading the aggregate table alone: each row only counts tasks where both compared conditions completed an API episode. For qwen-27b, English full interface beats Russian full interface on matched tasks (10 wins to 0, p = 0.2%) and beats German full interface (8 to 0, p = 0.8%); the Armenian comparison is also significant (9 to 1, p = 2.1%), while Spanish and French are directionally similar but not significant at this sample size. For gpt-oss-20b, matched samples are small and mostly “both failed”, so most comparisons do not reach significance, with the exception of the German full-interface vs. German-goal comparison (0 to 6 the other way, p = 3.1%).
| Model | Comparison | Matched tasks | Left only | Right only | Both solved | Both failed | Exact p |
|---|---|---|---|---|---|---|---|
| gpt-oss-20b | English full vs. Russian full | 35 | 1 | 2 | 0 | 32 | 100.0% |
| gpt-oss-20b | English full vs. Spanish full | 36 | 1 | 0 | 0 | 35 | 100.0% |
| gpt-oss-20b | English full vs. German full | 36 | 1 | 1 | 0 | 34 | 100.0% |
| gpt-oss-20b | English full vs. French full | 36 | 1 | 1 | 0 | 34 | 100.0% |
| gpt-oss-20b | English full vs. Armenian full | 35 | 1 | 3 | 0 | 31 | 62.5% |
| gpt-oss-20b | Spanish full vs. Spanish goal + English interface | 36 | 0 | 1 | 0 | 35 | 100.0% |
| gpt-oss-20b | German full vs. German goal + English interface | 36 | 1 | 0 | 0 | 35 | 100.0% |
| gpt-oss-20b | French full vs. French goal + English interface | 36 | 1 | 2 | 0 | 33 | 100.0% |
| gpt-oss-20b | Armenian full vs. Armenian goal + English interface | 35 | 2 | 0 | 1 | 32 | 50.0% |
| qwen-27b | English full vs. Russian full | 36 | 10 | 0 | 4 | 22 | 0.2% |
| qwen-27b | English full vs. Spanish full | 36 | 7 | 1 | 7 | 21 | 7.0% |
| qwen-27b | English full vs. German full | 36 | 8 | 0 | 6 | 22 | 0.8% |
| qwen-27b | English full vs. French full | 36 | 4 | 1 | 10 | 21 | 37.5% |
| qwen-27b | English full vs. Armenian full | 35 | 9 | 1 | 5 | 20 | 2.1% |
| qwen-27b | Spanish full vs. Spanish goal + English interface | 36 | 2 | 5 | 6 | 23 | 45.3% |
| qwen-27b | German full vs. German goal + English interface | 36 | 0 | 6 | 6 | 24 | 3.1% |
| qwen-27b | French full vs. French goal + English interface | 36 | 2 | 4 | 9 | 21 | 68.8% |
| qwen-27b | Armenian full vs. Armenian goal + English interface | 35 | 2 | 1 | 4 | 28 | 100.0% |
Discussion
The results are most naturally read as a control-surface effect. A one-time foreign-language goal can often be handled when the operating interface remains English. Changing the ongoing interface is harder: the model must repeatedly map the current observation to the correct precondition-sensitive action, and failures compound over several steps.
Token efficiency should be read with care. Tokens-per-success is not only a tokenizer property — it combines prompt length, completion length, number of turns, and success rate. A condition can be expensive because the language uses more tokens, because the agent takes more steps, or because fewer episodes succeed.
Earlier static, multiple-choice-style tasks remain part of the journey, but not the main evidence here. They helped harden scoring and counterbalancing, but did not stress the repeated observation-action loop. KeyBoxWorld is closer to the agentic setting we care about.
Limitations
This benchmark exposes language-surface effects in multi-step spatial tool use, but it should not be read as a universal ordering of languages or as a general comparison of model quality. The current multilingual extension is a partial matrix and must be repeated with fuller coverage and more models before the effect size can be treated as stable.
Incomplete matrix
The multilingual artifact has clean scored episodes, but not the planned full coverage — the run was interrupted by an API-level key-total-limit error before the full 3,840-episode matrix completed. Matched-pair rows reduce this problem but do not remove it.
Model coverage
This should not be read as a property of all LLMs. The extension adds a second model family (gpt-oss-20b alongside qwen-27b), but the matrix is still partial and not a leaderboard.
Hand-written surfaces
Hand-written templates make conditions controlled, but phrasing quality can still vary across languages. The runner now supports prompt variants; the next scored matrix should execute those variants rather than relying on one phrasing per language.
Symbolic IDs
Object IDs are intentionally language-neutral. That isolates the interface language, but real tasks often localize object names too.
Conclusion and future work
The main conclusion is that interface language is not just a localization detail for spatial agents. In this benchmark, changing the language of the operating surface changes how the model acts in the same hidden world. The result motivates treating interface language as part of the experimental design whenever LLM agents are evaluated through tools, observations, or simulator feedback.