smg

SMG v1.14.0-rc.2

1065 commits since v1.13.0 on release/1.14 (the merged pull requests, by type):

Features

• feat(gateway): Symphony is the default parser; --parser legacy and the X-SMG-Parser header keep the older parsers (#4677)
• feat(gateway): TokenSpeed engines serve every role on the direct ZMQ lane (#4649)
• feat(symphony): Llama 3.1 and 3.3 Instruct read with the json table under their own ids (#4655)
• feat(engine_zmq_adapter): a TokenSpeed engine's prompt logprobs, top-k candidates and abort status are read by the column names it declares (#4630)
• feat(gateway): media over the SGLang direct-ZMQ wire is assembled as the gRPC path assembles it, and the ZMQ lane runs the multimodal module (#4580)
• feat(tau2): add shared budget reservations (#4217)
• feat(servicer): a Rust servicer for the TensorRT-LLM gRPC contract, with its binding and launcher (#4494)
• feat(mock-worker): select repeatable output from prompt tokens (#4451)
• feat(servicer): the TokenSpeed servicer's control RPCs ride the wire's control call when the scheduler declares its reply (#4453)
• feat(servicer): the Rust SGLang servicer serves the EPD language side (#4413)
• feat(servicer): the Rust SGLang servicer carries multimodal inputs to the scheduler (#4405)
• feat(health): the readiness body names each served model with its readiness and a reason (#4412)
• feat(servicer): the Rust SGLang servicer serves prefill/decode disaggregation (#4411)
• feat(metrics): first-token and per-output-token latency histograms with their own bucket edges (#4410)
• feat(servicer): the Rust TokenSpeed servicer serves multimodal items of the Qwen3.5 MRoPE lineage (#4415)
• feat(servicer): the Rust TokenSpeed servicer serves prefill/decode disaggregation (#4408)
• feat(observability): the media pipeline's stages, items and fetches are series on the router and the worker side (#4377)
• feat(metrics): the router error counter carries the reason behind each error class (#4345)
• feat(helm): router.discovery.provider selects the worker discovery provider, and the Role exists only for what reads the Kubernetes API (#4391)
• feat(metrics): one per-worker routing-state series names why a worker is out of rotation (#4353)
• feat(metrics): the cached prompt tokens are a token type of the tokens counter and every cache-aware decision observes its match ratio (#4360)
• feat(metrics): the Symphony failure classes of a served model's table exist at zero from its first registration (#4348)
• feat(servicer): StartProfile and StopProfile on the vLLM wire, both servicers (#4369)
• feat(metrics): the KV capacity behind the engine's usage ratio and the block size the index routes by are gauges (#4357)
• feat(helm): a dashboard over the emitted families and a PrometheusRule with ten serving alerts (#4343)
• feat(helm): a gRPC worker can serve the servicer's Prometheus metrics from the engine pod (#4341)
• feat(mock-worker): the mock gRPC worker hosts the standard grpc.health.v1 service (#4338)
• feat(gateway): configure PD readiness capacity thresholds (#4331)
• feat(multimodal): expose media fetch timeouts (#4330)
• feat(servicer): a /metrics endpoint with request, token and latency metrics on every wire (#4149)
• feat(discovery): a Consul provider that discovers a service's passing instances (#4137)
• feat(glm): add native GLM-5.3 rendering and align Flash media processing (#4036)
• feat(multimodal): Nemotron-H Omni images shrink to a request's token budget, as the engine's tiler shrinks them (#4120)
• feat(multimodal): make the Hugging Face Hub download an optional hf-hub feature (#4119)
• feat(cache_aware): opt-in spill pricing for long-session prefill pools (#4089)
• feat(e2e): a mock engine row in the harness, with the answers stated by the test (#4070)
• feat(mock-worker): scripted answers on every wire, from a keyed file or the admin API (#4066)
• feat(discovery): a Slurm provider that discovers the nodes of running jobs (#4013)
• feat(symphony): the DeepSeek R1 line reads

SMG v1.14.0-rc.1

1061 commits since v1.13.0 on release/1.14 (the merged pull requests, by type):

Features

• feat(gateway): Symphony is the default parser; --parser legacy and the X-SMG-Parser header keep the older parsers (#4677)
• feat(gateway): TokenSpeed engines serve every role on the direct ZMQ lane (#4649)
• feat(symphony): Llama 3.1 and 3.3 Instruct read with the json table under their own ids (#4655)
• feat(engine_zmq_adapter): a TokenSpeed engine's prompt logprobs, top-k candidates and abort status are read by the column names it declares (#4630)
• feat(gateway): media over the SGLang direct-ZMQ wire is assembled as the gRPC path assembles it, and the ZMQ lane runs the multimodal module (#4580)
• feat(tau2): add shared budget reservations (#4217)
• feat(servicer): a Rust servicer for the TensorRT-LLM gRPC contract, with its binding and launcher (#4494)
• feat(mock-worker): select repeatable output from prompt tokens (#4451)
• feat(servicer): the TokenSpeed servicer's control RPCs ride the wire's control call when the scheduler declares its reply (#4453)
• feat(servicer): the Rust SGLang servicer serves the EPD language side (#4413)
• feat(servicer): the Rust SGLang servicer carries multimodal inputs to the scheduler (#4405)
• feat(health): the readiness body names each served model with its readiness and a reason (#4412)
• feat(servicer): the Rust SGLang servicer serves prefill/decode disaggregation (#4411)
• feat(metrics): first-token and per-output-token latency histograms with their own bucket edges (#4410)
• feat(servicer): the Rust TokenSpeed servicer serves multimodal items of the Qwen3.5 MRoPE lineage (#4415)
• feat(servicer): the Rust TokenSpeed servicer serves prefill/decode disaggregation (#4408)
• feat(observability): the media pipeline's stages, items and fetches are series on the router and the worker side (#4377)
• feat(metrics): the router error counter carries the reason behind each error class (#4345)
• feat(helm): router.discovery.provider selects the worker discovery provider, and the Role exists only for what reads the Kubernetes API (#4391)
• feat(metrics): one per-worker routing-state series names why a worker is out of rotation (#4353)
• feat(metrics): the cached prompt tokens are a token type of the tokens counter and every cache-aware decision observes its match ratio (#4360)
• feat(metrics): the Symphony failure classes of a served model's table exist at zero from its first registration (#4348)
• feat(servicer): StartProfile and StopProfile on the vLLM wire, both servicers (#4369)
• feat(metrics): the KV capacity behind the engine's usage ratio and the block size the index routes by are gauges (#4357)
• feat(helm): a dashboard over the emitted families and a PrometheusRule with ten serving alerts (#4343)
• feat(helm): a gRPC worker can serve the servicer's Prometheus metrics from the engine pod (#4341)
• feat(mock-worker): the mock gRPC worker hosts the standard grpc.health.v1 service (#4338)
• feat(gateway): configure PD readiness capacity thresholds (#4331)
• feat(multimodal): expose media fetch timeouts (#4330)
• feat(servicer): a /metrics endpoint with request, token and latency metrics on every wire (#4149)
• feat(discovery): a Consul provider that discovers a service's passing instances (#4137)
• feat(glm): add native GLM-5.3 rendering and align Flash media processing (#4036)
• feat(multimodal): Nemotron-H Omni images shrink to a request's token budget, as the engine's tiler shrinks them (#4120)
• feat(multimodal): make the Hugging Face Hub download an optional hf-hub feature (#4119)
• feat(cache_aware): opt-in spill pricing for long-session prefill pools (#4089)
• feat(e2e): a mock engine row in the harness, with the answers stated by the test (#4070)
• feat(mock-worker): scripted answers on every wire, from a keyed file or the admin API (#4066)
• feat(discovery): a Slurm provider that discovers the nodes of running jobs (#4013)
• feat(symphony): the DeepSeek R1 line reads

Shepherd Model Gateway v1.13.0

SMG v1.13.0 makes the gRPC serving path faster, lighter and closer to the engines behind it. On the same 18 engines as v1.12.0, it serves nearly twice the image traffic and finishes those requests 3.3× sooner, with 38–56% less CPU per request and about half the router memory. It also serves 1.63× the image traffic of another open-source router on the same hosts. The tokenizer, media pipeline and chat templates now produce what the engines' own processors produce, and the OpenAI and Anthropic routes answer the way the public APIs do. The gateway refuses misconfiguration and overload up front instead of failing later, and this release closes several security gaps, listed first under Notable fixes. Symphony, a single table-driven parser for reasoning and tool calls, now runs inside the gateway as an opt-in beta.

This release includes 567 commits from 11 contributors, including 2 first-time contributors. Some defaults changed, and some deployments need a step before upgrading: see Before you upgrade.

The fleet figures come from this release's validation run, which compared v1.12.0 and v1.13.0 on the same hosts, engines and traffic. Other figures come from benchmarks of the individual changes. They measure different workloads and should not be combined into one overall speedup. For latency, p99 means 99% of measurements were at or below that value.

Before you upgrade

• Postgres or Oracle history store: this release adds schema migration 4. Set auto_migrate: true in the store's schema config (or DB_AUTO_MIGRATE=true), or apply the statements the gateway prints, and create the sweep index on existing tables (see Storage).
• Browser clients: CORS is closed unless --cors-allowed-origins is set. Pass '*' to keep v1.12.0's open policy.
• Media from internal hosts: URLs that resolve to private or loopback addresses are refused. Set --mm-allow-private-media-hosts on the gateway, and SMG_MM_ALLOW_PRIVATE_MEDIA_HOSTS=1 on workers that run SMG's media pipeline.
• Helm: the readiness probe moved to /readiness, so a router whose workers register only through the admin API never becomes Ready; set router.readinessProbe.path: /health for that setup. A custom router image must run as UID 65532 or override router.securityContext.runAsUser.
• Stricter start-up: an unreachable Postgres or Redis store, a mesh that cannot start, and a list of malformed settings now stop the gateway before it binds a port. Check your configuration against Configuration that now refuses to start.
• Engine pods on the Rust servicer: roll gateways first, then install smg 1.13.0 together with smg-grpc-servicer 0.14.0. The new servicer fails with TypeError against the 1.12.0 smg wheel.
• Python Router(...) users: metrics now bind all interfaces when prometheus_host is unset (v1.12.0 bound 127.0.0.1).
• Dashboards and API clients: two metrics became histograms, error labels and error envelopes changed, and some requests that used to pass are now refused with 400. See Upgrade notes.

Highlights

Nearly twice the image traffic on the same GPUs, at less than half the CPU per request

v1.12.0 and v1.13.0 ran in turn on the same 18 GPU hosts, with the same vLLM 0.31.0 engines and the same traffic:

• Text traffic is paced by the engines, so its throughput holds. The gain there is cost

Shepherd Model Gateway v1.12.0

SMG v1.12.0 adds AMD prefill/decode support contributed by the AMD team and improves performance across gateway streaming, tokenization, engine services, cache-aware routing, and multimodal processing. It also adds Decisions and System One endpoints and fixes streaming, tool-call, and worker-management issues.

This release includes 141 commits from 19 contributors, including 9 first-time contributors.

Benchmarks below are reported in the linked PRs. They measure different workloads and should not be combined into one overall speedup. For latency, p99 means 99% of measurements were at or below that value.

Highlights

AMD prefill/decode support — contributed by the AMD team

The AMD team added support for running prompt processing (prefill) and token generation (decode) on separate AMD GPU workers, using vLLM and MoRI-IO. Thanks to @chunfangamd for the implementation, follow-up fixes, and hardware validation.

• READ and WRITE cache transfers. SMG coordinates both modes between vLLM HTTP workers, fixing the missing transfers that previously caused incorrect output or hanging requests. Streaming and non-streaming /v1/chat/completions and /v1/completions are supported. #2732.
• Data-parallel deployments. SMG assigns each request to the same data-parallel rank on both workers, distributes requests across ranks, and rejects mismatched configurations. This enables models such as DeepSeek-R1 with multiple data-parallel ranks. #2798.
• Optional concurrent WRITE dispatch. Starting both workers together lets the decode worker allocate cache while prefill runs, reducing dispatch delay. Set moriio_write_dispatch=concurrent on the decode worker to enable it. Sequential dispatch remains the default because early cancellation can leave cache memory allocated in the tested vLLM connector. #2742.
• Validated transfers. SMG checks the transfer identity, peer details, ports, cache-block lists, and rank information before accepting a transfer. If validation fails, a sequential request stops before decode begins; a concurrent request fails before its decode response is returned. These checks protect against hung requests, incorrect output, and engine crashes. #2764, #2773.

Validated on two AMD MI355X nodes, including Qwen3-0.6B with tensor-parallel sizes of 1 and 8, and DeepSeek-R1-0528 (FP8) with eight data-parallel ranks and expert parallelism. Tests checked actual cache data transferred over the network, streaming responses, concurrent requests, and failure handling. Validation.

In the reported Qwen3-0.6B test, concurrent WRITE reduced median time to first token from 71.5–72.0 ms to 46.1–46.2 ms at four concurrent requests—about 36% lower—and increased throughput from 9.89–9.98 to 10.62–10.65 requests/s. These measurements used the same two-node setup, approximately 4,350 input tokens and 64 output tokens, with three alternating rounds. Benchmark.

Faster gateway streaming and tokenization

• Less CPU per streamed request. Logging, request counting, router lookup, and cache-aware worker selection avoid repeated locking and bookkeeping. Ordinary log-level settings use the faster filter automatically; advanced span filters keep their existing behavior. #2752.
• Faster follow-up turns in long conversations. The tokenizer now caches the deepest reusable boundary after a cache hit, so each new turn can reuse the growing conversation instead of repeatedly tokenizing its earlier text. Cache storage remains proportional to prompt length.