← All lab notesREPORT · 2026-09-19

Does context make AI analytics more efficient?

We asked an AI agent to build dashboards on the same data, with and without a data catalog and glossary. Median token use fell 14.2%, and passes rose from 77.8% to 96.3%.

↳ 14.2% fewer median tokens with a catalog + glossary

We asked an AI agent to build dashboards for a broad set of business tasks using the same ClickHouse data, both with and without a data catalog and glossary. We then compared its token use and checked the results.

For all tests we used DeepSeek V4 Flash 0731, chosen for its balance of efficiency and performance in agent workflows.

The question

A database can tell an agent which columns exist. A business glossary can explain what those columns mean together: which refunds count, how annual contracts become monthly revenue, and which clock determines a service commitment.

That context has a price. The agent receives extra tool descriptions, reads metadata, and carries those results through subsequent model calls. The experiment asks whether the added context reduces enough discovery and correction work to offset the extra tokens.

Experiment design

Held constantDatasource · tasks · model · starting state
One interventionData catalog + glossary + domain context
Primary measureReported model input + output tokens

A — Datasource only

Explore the data: inspect tables and sample rows, work out how the data connects, calculate the requested metrics, and save the dashboard.

  • Inspect data
  • Calculate metrics
  • Build the dashboard

B — With catalog + glossary

Add business context first: find relevant datasets by business domain, read metric definitions, and check documented relationships. Then calculate the metrics and save the dashboard.

  • Find relevant data
  • Read definitions
  • Check relationships

The business dataset is entirely generated: 2,089 rows, six tables, three business domains, seventeen glossary terms, and eighteen independently checked analytical facts. Unequal queue volumes, cross-month refunds, and changing subscription snapshots expose plausible calculation mistakes. These are controlled tasks on a small schema, not customer workloads.

Business domains tested

DomainTablesPrimary metric
Commerce
Net sales, refund amount rates, and net order value. Preserve order grain and attribute later refunds correctly.
orders ↔ refundsNet Sales
Customer growth
Recurring revenue, active seats, annualized value, and signed revenue changes. Normalize prices and keep snapshots separate.
subscription_snapshots ↔ plan_pricesMonthly Recurring Revenue
Service operations
SLA attainment, breach volumes, and resolution-time distributions. Respect queue targets, unequal volumes, and percentile definitions.
tickets ↔ queue_targetsResolution SLA Attainment

Results

ConditionPass rate
Datasource only77.8%
With catalog + glossary96.3%

Across the comparable token measurements in our test set, the catalog used fewer tokens in 68.0% of comparisons. The median paired token change was −14.2%.

Input accounted for 98.2% of reported tokens in our test set. Tool responses and the accumulated conversation are processed again as the agent takes further steps; cached input still counts. A catalog adds context, so a shorter final answer alone cannot demonstrate token savings.

How we evaluated correctness

Our platform includes a grader that evaluates the agent’s actions and the results it produces. We use this assessment alongside token consumption to compare the two approaches. Each task asks for two persistent views; reference SQL was checked against independently calculated results. The verifier reopens saved dashboards and replays their actual bound queries, checking numbers, saved chart configuration, durable completion, and ownership. A catalog lookup alone does not establish correctness.

Reading these results

Count the whole task

Token volume is the runtime- or provider-reported input plus output, counted once per terminal turn. Cached input remains in the total; reasoning is already included in output. Review, scoring, and setup are outside task token totals. Tokens are not a dollar-cost estimate.

Hold the starting point steady

Both conditions used the same data, tasks, model, and starting state. The difference was access to a data catalog and glossary. Each run started in a fresh session.

Verify the product

Percentage comparisons accept equivalent proportions (0.52) or percentage values (52); they do not verify displayed percentage labels. Presentation is reviewed separately.

Keep the claim proportional

This is a small controlled study of one model revision and a synthetic warehouse with six business tables. The intervention combines catalog, glossary, descriptions, and domains; it does not isolate the glossary's individual effect. Larger schemas and underspecified requests remain untested.

More from the lab