Open Data · A dataset hub for LLM serving research

Datasets

Released datasets

01 / Chutes · One year of requests

One Year in LLM Serving

Released

A year of production request traces metadata from Chutes, covering model usage, token counts, and prefix-cache reuse.

Hugging Face dataset Code and docs Paper

6.12Brequests
9,174models
1 yearof traffic

Research uses

Benchmark batching and scheduling with recorded request arrivals and input/output token lengths.
Reconstruct user sessions from request metadata to analyze usage patterns and behavior.
Explore GPU colocation for low-traffic models using their busy and idle periods.

Schema

Dataset documentation ↗

Chutes schema
Field	Type
`request_id`	int
`function_name`	string
`chute_id` · `instance_id`	string
`user_id`	string
`started_at` · `completed_at`	timestamp
`input_tokens` · `output_tokens` · `cache_tokens`	int
`ttft`	float

user_id values rotate every three months, so one person appears under different IDs across the year.

Findings

01
Requests exhibit high temporal locality. 99% of repeat requests from the same user to the same model arrive within 15 minutes.

Interarrival Time
02
LRU often matches or outperforms more complex cache eviction policies.

Cache eviction policies
03
Cache-aware routing raises the token hit ratio, with 5–7% load imbalance in simulation.

Cache-aware routing

03 / FreeInference · Infrastructure metrics

GPU and serving telemetry

Released

Live time series on GPU utilization, memory, and inference-engine performance from the FreeInference fleet, connecting infrastructure behavior with serving demand.

Live telemetry dashboard Snapshot API

Contents: GPU, inference-engine, and host time series
Source: FreeInference serving fleet, via DCGM and engine metrics
Resolution: 1-minute samples, rolled up to 20-minute and hourly

Research uses

Study GPU utilization and capacity needs as serving demand changes.
Identify bottlenecks in memory, KV-cache capacity, and request queues.
Analyze how infrastructure load relates to prefill and decode throughput.
Measure speculative-decoding accept rates and hierarchical KV-cache traffic under live load.

Schema

Field reference ↗

GPU and serving telemetry schema
Field	Type
gpu · one row per GPU per sample
`ts`	int
`node_id` · `gpu_id`	string
`utilization_pct` · `memory_util_pct` · `temperature_c` · `clocks_sm_mhz`	int
`memory_pct` · `power_w`	float
`tensor_active` · `dram_active`	float
`nvlink_tx_bps` · `nvlink_rx_bps` · `pcie_tx_bps` · `pcie_rx_bps`	float
sglang · one row per server per sample
`server_id` · `name`	string
`running` · `queued`	int
`used_tokens` · `max_tokens` · `active_tokens` · `waiting_tokens`	int
`token_usage` · `cache_hit_rate`	float
`prefill_tps` · `cached_prefill_tps` · `gen_throughput`	float
`ttft_avg_s` · `queue_avg_s` · `prefill_avg_s` · `tpot_avg_s` · `e2e_latency_avg_s`	float
`spec_accept_length` · `spec_accept_rate`	float
cpu · one row per node per sample
`cpu_util_pct` · `load1` · `load5`	float
`mem_used_mb` · `mem_total_mb` · `mem_available_mb`	int

ts is epoch milliseconds. token_usage, cache_hit_rate, tensor_active, and dram_active are fractions in 0–1, not percentages. memory_pct is VRAM occupancy; memory_util_pct is memory-bandwidth utilization. HiCache tiering, request shape, and error counters are in the field reference.

Access

The dashboard serves the current snapshot at /api/dashboard and history at /api/history?hours=N, where N is 2 for 1-minute samples, 24 for 20-minute averages, or 168 for hourly averages. Both are open, and each chart also downloads as CSV. Bulk export of up to 30 days of hourly rollups is password-protected, so write to data@freeinference.org for a copy.

04 / FreeInference · KV-cache traces

FreeInference agent-serving dataset

Released

Production agent traces from FreeInference across 13 diverse agent harnesses, covering prefix-block hashes, request timing, and sessions for studying KV-cache reuse and agent serving.

Explore the trace Hugging Face dataset Code and docs Paper

16 weeksof traffic
12Ksessions
209Binput tokens

Research uses

Replay prefix-block hashes to evaluate KV-cache eviction policies, cache sizes, and prefix reuse.
Benchmark scheduling and batching using recorded request arrivals and serving latencies.
Analyze context mutations and harness-specific behavior to understand their impact on cache reuse.
Study inter-arrival times and tool latencies to guide KV-cache offloading, eviction, and prefetching.
Export traces in LLMLCS format for caching experiments with libCacheSim.

Schema

Dataset documentation ↗

FreeInference agentic trace schema
Field	Type
trace · one row per session
`week` · `day`	string
`session_id` · `user_id` · `harness`	string
`start_ms` · `end_ms`	int
`requests`	list[object]
requests · one per LLM request
`step` · `turn_index` · `n_messages`	int
`step_trigger` · `link_reason` · `finish_reason`	string
`start_ms` · `end_ms` · `ttft_ms`	int
`model` · `provider`	string
`stream`	bool
`provider_tokens` · `role_tokens` · `mutation`	object
`tool_definition_names`	list[string]
`block_ids`	list[int]
`tool_calls`	list[object]
tool_calls · one per tool call
`call_index` · `result_tokens` · `latency_ms`	int
`name` · `category` · `parse_status` · `outcome`	string
`programs`	list[string]
`arguments`	list[list[string]]
`subagent`	object

FreeInference agentic trace example values
Field	Example
trace · one row per session
`week` · `day`	`"2026-05-17_2026-05-23"` · `"2026-05-17"`
`session_id` · `user_id` · `harness`	`"..."` · `"..."` · `"claude-code"`
`start_ms` · `end_ms`	`1778977459236` · `1778984067198`
`requests`	`[{…}, {…}, …]`
requests · one per LLM request
`step` · `turn_index` · `n_messages`	`0` · `0` · `179`
`step_trigger` · `link_reason` · `finish_reason`	`"user"` · `null` · `"tool_calls"`
`start_ms` · `end_ms` · `ttft_ms`	`1778977459236` · `1778977468828` · `7889`
`model` · `provider`	`"glm-5.1"` · `"B"`
`stream`	`true`
`provider_tokens`	`{"prompt": 52731, "completion": 230, "cached": 1280}`
`role_tokens`	`{"system": 1790, "user": 1110, "assistant": 15916, "tool": 25725, "tool_definitions": 8638}`
`mutation`	`{"transition": "append", "cause": null}`
`tool_definition_names`	`["read_file", "write_file", "terminal", "execute_code"]`
`block_ids`	`[123, 456, 789, "..."]`
`tool_calls`	`[{…}]`
tool_calls · one per tool call
`call_index` · `result_tokens` · `latency_ms`	`0` · `842` · `5272`
`name` · `category` · `parse_status` · `outcome`	`"terminal"` · `"*Interpreter"` · `"success"` · `"error"`
`programs`	`["python"]`
`arguments`	`[["command", "python <path> --status"], ["timeout", "<num>"]]`
`subagent`	`null`

Findings

01
Input and output price does not matter for your agent (cache read price dominates). Cached tokens dominate the proportion: in 72% of sessions, cached input is the biggest cost.

Cost breakdown by Session

Tokens priced 1 : 10 : 50 — cached input, uncached input, output.
02
Tool execution is the bottleneck. Halving tool time speeds a session up by 1.38×, against 1.09× for prefill and 1.16× for decode.

Speedup by component

Each tool gap capped at 15 minutes; user idle time excluded.

03

Context mutations are common. Although most requests are compliant, 4.00% of requests mutate prior context, breaking the prefix and causing significant fresh prefill.

Context mutations by cause

Share of steps and of fresh prefill, by transition type
Transition	Steps	Fresh prefill
Append	96.00%	37.5%
Mutation	4.00%	62.5%
Mutation by cause
Injection	0.96%	7.4%
System prompt	0.74%	26.9%
Tool definitions	0.40%	13.9%
Dropped turns	0.36%	5.3%
Compaction	0.04%	0.6%
Tool result drop	0.51%	3.2%
Other	1.00%	5.2%

Fresh prefill is measured under infinite KV-cache retention, so it captures reuse lost to context changes rather than to eviction.

Release plan

Prompt and response text is not included; it is planned for the raw prompt dataset. Tool calls are sanitized, and request inputs are tokenized with o200k_base into 16-token prefix blocks.

Roadmap

Upcoming datasets

02 / Chutes · Conversational text

Chutes conversational dataset

Planned

Prompts and model responses collected from an optional Chutes research endpoint. Traffic is mixed, but tends to be conversational and roleplay rather than agentic tool calling.

Contents: Chat and roleplay prompts and responses
Source: Optional Chutes research endpoint
Focus: Conversational workloads and prefix reuse

Research uses

Compare conversational and roleplay traffic against agentic tool-use workloads.
Measure prefix reuse across the conversation history carried in each request.
Study how chat and character prompts are constructed and revised across turns.
Build serving benchmarks grounded in real open-ended conversational prompts.

Planned schema

Chutes conversational schema
Field	Type
`request_id`	string
`model_id`	string
`messages`	list[object]
`response`	object
`started_at` · `completed_at`	timestamp
`prompt_tokens` · `completion_tokens` · `cached_tokens`	int

Release plan

Conversational prompts and responses from the optional research endpoint, released with field descriptions.

05 / FreeInference · Agent sessions

FreeInference raw prompt dataset

Planned

Sanitized agent conversations from FreeInference, covering user prompts, model responses, and tool calls, for trajectory analysis and model training.

Contents: Prompts, responses, and tool calls
Source: FreeInference agent traffic
Focus: Agent behavior and model training

Research uses

Analyze agent trajectories, including user prompts, tool selection, and tool results where recorded.
Study how agents construct prompts, revise them, and carry conversation history across turns.
Build evaluation sets and serving benchmarks grounded in real agent tasks.
Fine-tune models on production agent trajectories to improve multi-turn tool use.
Train draft models for speculative decoding on agent prompts paired with the target model’s responses.

Planned schema

FreeInference raw prompt schema
Field	Type
`request_id` · `session_id` · `model_id`	string
`messages`	list[object]
`response`	object
`tool_calls` · `tool_results`	list[object]
`started_at` · `completed_at`	timestamp
`usage`	object

Release plan

Sanitized conversations will be released separately from the agentic trace, pending IRB approval.

Team

Contributors

Harvard MadSys and collaborators.

About

Open data for agentic AI systems

We study LLM and agent serving to improve efficiency and broaden access to AI. This site shares datasets and analysis code from Harvard MadSys and collaborators.

data@freeinference.org Harvard MadSys on GitHub FreeInference

Citation

Cite this data

If these datasets support your work, please cite A Year in LLM Serving: Workload Evolution, Caching and Load-Balancing, which introduces the released one-year Chutes trace. For the FreeInference agentic trace, cite From Requests to Sessions: A Large-Scale Characterization of Human-Driven Agentic Workloads. Datasets still in the roadmap will carry their own citation in the GitHub artifact released alongside each one.

BibTeX · One Year in LLM Serving

@misc{nixon2026year,
  title         = {A Year in LLM Serving: Workload Evolution, Caching
                   and Load-Balancing},
  author        = {Nixon, William and Durbin, Jon and Standhartinger, Florian
                   and Gunawi, Haryadi S. and Yang, Juncheng},
  year          = {2026},
  eprint        = {2608.13573},
  archivePrefix = {arXiv},
  primaryClass  = {cs.AI},
  url           = {https://arxiv.org/abs/2608.13573}
}

BibTeX · FreeInference agentic trace

@article{freeinference2026requests,
  title  = {From Requests to Sessions: A Large-Scale Characterization
            of Human-Driven Agentic Workloads},
  author = {Nixon, William and Tian, Muxin and Zheng, Yunjia
            and Gunawi, Haryadi S. and Yang, Juncheng},
  year   = {2026}
}

A dataset hub for
LLM serving research.

Dataset catalog

Chutes 02

FreeInference 03

Released datasets

One Year in LLM Serving

Research uses

Schema

Findings

GPU and serving telemetry

Research uses

Schema

Access

FreeInference agent-serving dataset

Research uses

Schema

Findings

Release plan

Upcoming datasets

Chutes conversational dataset

Research uses

Planned schema

Release plan

FreeInference raw prompt dataset

Research uses

Planned schema

Release plan

Contributors

Open data for agentic AI systems

Cite this data

A dataset hub for LLM serving research.

Dataset catalog

Chutes 02

FreeInference 03

One Year in LLM Serving

Research uses

Schema

Findings

GPU and serving telemetry

Research uses

Schema

Access

FreeInference agent-serving dataset

Research uses

Schema

Findings

Release plan

Upcoming datasets

Chutes conversational dataset

Research uses

Planned schema

Release plan

FreeInference raw prompt dataset

Research uses

Planned schema

Release plan

Contributors

Open data for agentic AI systems

Cite this data

A dataset hub for
LLM serving research.