# GPU and serving telemetry — field reference

Fields as served by the OpenInfra dashboard at <https://openinfra.freeinference.org>, verified
against the live endpoints on 2026-09-19. The feed is live infrastructure telemetry, so the fleet
and the deployments it covers change over time; the field names below are what the API returns today.

Values are `null` wherever an exporter did not report the metric — every numeric field below is
nullable, including ones marked `int`. Metrics come from the node exporter (host), NVML and DCGM
(GPU), and the engine's own metrics endpoint (SGLang or vLLM).

## Endpoints

| endpoint | access | contents |
|---|---|---|
| `/api/dashboard` | open | Current snapshot: `kpis`, `nodes`, `gpus`, `sglang`, `served`, `requests`. Cached for 5 minutes. |
| `/api/history?hours=N` | open | Time series: `gpu`, `sglang`, `cpu`. `N` is 2 (1-minute samples), 24 (20-minute averages), or 168 (hourly averages). `grain_ms` reports the grain in use. |
| `/api/export?days=N` | password | Up to 30 days of hourly rollups, plus the current snapshot and served totals. |

Each chart on the dashboard also exports as CSV from its zoom view, without a password.

## Conventions

- `ts`, `last_seen_at`, `created_at`, and `generated_at` are milliseconds since the epoch.
- `token_usage`, `cache_hit_rate`, `tensor_active`, `dram_active`, and `spec_accept_rate` are
  fractions in 0–1, not percentages. Fields ending in `_pct` are percentages in 0–100.
- `*_tps` fields are tokens per second; the dashboard renders them per minute.
- `*_avg_s` fields are seconds, averaged over the sample window. `tpot_avg_s` is seconds per output
  token; the dashboard shows it as inter-token latency in milliseconds.
- `*_bps` fields are bytes per second.
- GPU memory has two distinct fields: `memory_pct` (and `memory_used_mb` / `memory_total_mb`) is how
  much VRAM is occupied, while `memory_util_pct` is memory-bandwidth utilization — the share of time
  device memory was being read or written. Likewise `utilization_pct` is SM utilization, not memory.

## History: `gpu`

One row per GPU per sample.

| column | type | meaning |
|---|---|---|
| `ts` | `int` | Sample time. |
| `node_id` | `string` | Host the GPU belongs to, e.g. `h200a`. |
| `gpu_index` | `int` | Index of the GPU within its host. |
| `gpu_id` | `string` | `node_id:gpu_index`, e.g. `h200a:0`. The row key together with `ts`. |
| `utilization_pct` | `int` | SM utilization, 0–100. |
| `memory_pct` | `float` | VRAM occupied, 0–100. |
| `memory_util_pct` | `int` | Memory-bandwidth utilization, 0–100. |
| `temperature_c` | `int` | GPU temperature in Celsius. |
| `power_w` | `float` | Board power draw in watts. |
| `clocks_sm_mhz` | `int` | SM clock in MHz. |
| `tensor_active` | `float` | DCGM tensor-pipe active fraction, 0–1. |
| `dram_active` | `float` | DCGM DRAM (HBM) active fraction, 0–1. |
| `nvlink_tx_bps` | `float` | NVLink bytes transmitted per second. |
| `nvlink_rx_bps` | `float` | NVLink bytes received per second. |
| `pcie_tx_bps` | `float` | PCIe bytes transmitted per second. |
| `pcie_rx_bps` | `float` | PCIe bytes received per second. |

## History: `sglang`

One row per serving deployment per sample. The series is named for SGLang but also carries vLLM
deployments; the snapshot's `engine` field says which.

| column | type | meaning |
|---|---|---|
| `ts` | `int` | Sample time. |
| `node_id` | `string` | Host running the deployment. |
| `server_id` | `string` | Deployment id, e.g. `h200a-replica-a`. The row key together with `ts`. |
| `name` | `string` | Deployment label shown on the dashboard, e.g. `deepseek-V4-a`. Distinct from the served model, which the snapshot reports as `model_name`. |
| `running` | `int` | Requests currently being served. |
| `queued` | `int` | Requests waiting to be scheduled. |
| `req_per_min` | `float` | Requests completed per minute. |
| `used_tokens` | `int` | KV-cache tokens in use. |
| `max_tokens` | `int` | KV-cache capacity in tokens. |
| `active_tokens` | `int` | Tokens belonging to running requests. |
| `waiting_tokens` | `int` | Uncached prompt tokens queued for prefill. |
| `token_usage` | `float` | KV-cache fill, `used_tokens / max_tokens`, 0–1. |
| `cache_hit_rate` | `float` | Prefix-cache hit rate, 0–1. |
| `prefill_tps` | `float` | Uncached prompt tokens prefilled per second. |
| `cached_prefill_tps` | `float` | Prompt tokens served from cache per second. |
| `gen_throughput` | `float` | Decode tokens generated per second. |
| `ttft_avg_s` | `float` | Mean time to first token. |
| `queue_avg_s` | `float` | Mean time spent queued, the first component of TTFT. |
| `prefill_avg_s` | `float` | Mean prefill time, the second component of TTFT. |
| `chunked_prefill_avg_s` | `float` | Mean time per chunked-prefill step. |
| `tpot_avg_s` | `float` | Mean time per output token. |
| `e2e_latency_avg_s` | `float` | Mean end-to-end request latency. |
| `spec_accept_length` | `float` | Mean accepted draft tokens per speculative-decoding step. |
| `spec_accept_rate` | `float` | Share of draft tokens accepted, 0–1. |
| `sm_util_pct` | `int` | SM utilization of the GPUs backing this deployment, 0–100. |
| `sglang_util_pct` | `int` | Engine-reported utilization, 0–100. |
| `hicache_host_used` | `int` | Host-tier KV cache in use, in tokens. |
| `hicache_host_total` | `int` | Host-tier KV cache capacity, in tokens. |
| `device_cached_tps` | `float` | Cached tokens served from device memory per second. |
| `host_cached_tps` | `float` | Cached tokens served from the host tier per second. |
| `loadback_tps` | `float` | Tokens loaded back from host to device per second. |
| `evict_tps` | `float` | Tokens evicted from the device cache per second. |
| `backup_tps` | `float` | Tokens backed up from device to host per second. |
| `prompt_tokens_avg` | `float` | Mean prompt length of completed requests, in tokens. |
| `uncached_prompt_tokens_avg` | `float` | Mean uncached part of the prompt, in tokens. |
| `generation_tokens_avg` | `float` | Mean generated length, in tokens. |
| `http_error_per_min` | `int` | 4xx and 5xx responses per minute. |
| `retracts` | `int` | Requests retracted when the KV cache ran out. |

## History: `cpu`

One row per host per sample.

| column | type | meaning |
|---|---|---|
| `ts` | `int` | Sample time. |
| `node_id` | `string` | Host id. The row key together with `ts`. |
| `cpu_util_pct` | `float` | CPU utilization, 0–100. |
| `load1` | `float` | 1-minute load average. |
| `load5` | `float` | 5-minute load average. |
| `mem_used_mb` | `int` | System memory in use, in MB. |
| `mem_total_mb` | `int` | System memory installed, in MB. |
| `mem_available_mb` | `int` | System memory available, in MB. |

## Snapshot-only fields

`/api/dashboard` repeats the fields above under different names in places, and adds inventory that
the history series leaves out.

### `nodes`

`id`, `hostname`, `region`, `cluster`, `status` (`string`); `cpu_count`, `gpu_count`,
`procs_running`, `procs_total`, `swap_used_mb`, `last_seen_at`, `created_at` (`int`); `load15`
(`float`); `avg_utilization_pct` (`float`); `metadata` (`object`, carrying the scrape counters
`scrapes`, `scrape_hz`, `window_s`); plus the `cpu` history fields.

### `gpus`

`id` (the `gpu_id`), `node_id`, `hostname`, `region`, `cluster`, `uuid`, `driver`, `pstate`,
`status` (`string`); `name` (`string`, the GPU model, e.g. `NVIDIA H200`); `memory_used_mb`,
`memory_total_mb`, `power_limit_w`, `fan_speed_pct`, `clocks_mem_mhz`, `ecc_errors`,
`last_seen_at` (`int`); `processes` (`list`); plus the `gpu` history fields.

### `sglang`

`id` (the `server_id`), `node_id`, `hostname`, `region`, `status` (`string`); `model_name`
(`string`, the served model, e.g. `deepseek-v4-flash`); `endpoint` (`string`); `engine` (`string`,
`sglang` or `vllm`); `gpu_indexes` (`string`, comma-separated); `max_running_requests`,
`prompt_tokens_total`, `generation_tokens_total`, `last_seen_at` (`int`); `weight_gb`,
`kv_cache_gb` (`float`). The counter names are longer here than in the history series:
`num_running_reqs`, `num_queue_reqs`, `num_used_tokens`, `max_total_num_tokens`,
`num_active_tokens`, and `num_waiting_uncached_tokens` correspond to `running`, `queued`,
`used_tokens`, `max_tokens`, `active_tokens`, and `waiting_tokens`.

### `served`

Requests and tokens delivered over the last hour and the last 24 hours, fleet-wide and per
deployment: `reqs`, `cache_tokens`, `prefill_tokens`, `decode_tokens`.

### `requests`

A rolling sample of recent individual requests — the only per-request rows in the feed.

| column | type | meaning |
|---|---|---|
| `ts` | `int` | When the request was recorded. |
| `server_id` | `string` | Deployment that served it. |
| `model` | `string` | Model served. |
| `input_len` | `int` | Prompt tokens. |
| `cache_len` | `int` | Prompt tokens served from cache. |
| `uncached_len` | `int` | Prompt tokens that had to be prefilled. |
| `output_len` | `int` | Generated tokens. |
| `queue_s` | `float` | Seconds queued before scheduling. |
| `prefill_s` | `float` | Seconds spent in prefill. |
| `ttft_s` | `float` | Time to first token, in seconds. |
| `tpot_s` | `float` | Seconds per output token. |
| `e2e_s` | `float` | End-to-end latency, in seconds. |
| `decode_tps` | `float` | Decode tokens per second for this request. |
| `spec_accept_len` | `float` | Accepted draft tokens per speculative step. |
| `spec_accept_rate` | `float` | Share of draft tokens accepted, 0–1. |
| `batch_size` | `int` | Batch the request was decoded in. |

### `kpis`

Fleet rollups of the above, computed across online nodes, GPUs, and deployments: `nodes`,
`nodes_online`, `gpus`, `gpus_online`, `sglang_servers`, `sglang_online`, `avg_utilization_pct`,
`avg_temperature_c`, `power_w`, `memory_used_mb`, `memory_total_mb`, `tensor_active`,
`dram_active`, `nvlink_gbps`, `running_reqs`, `queue_reqs`, `gen_throughput`, `prefill_tps`,
`token_usage`, `cache_hit_rate`, `used_tokens`, `max_tokens`, `active_tokens`, `waiting_tokens`,
`cpu_util_pct`, `load1`, `ram_used_mb`, `ram_total_mb`.