sie

⭐ Help us reach more developers and grow the SIE community. Star this repo!

About

SIE is an open-source inference engine that runs the models behind every agent task through one API: search and retrieval, document-to-markdown conversion, structured output, content safety, the agent loop itself, and the translation, transcription, and vision calls around it. It replaces the patchwork of a separate model server per task with one system that serves 100+ models, loading each on demand.

• OpenAI-compatible API for drop-in migration: /v1/embeddings, /v1/chat/completions, /v1/completions, /v1/responses
• Pre-configured model catalog: Stella, SPLADE, Qwen3, GLiNER, SigLIP, and more; embedding and retrieval models benchmarked on MTEB
• Serves multiple models simultaneously with on-demand loading and LRU eviction
• Ships Kubernetes and Helm deployment configs for the load-balancing gateway, KEDA autoscaling, and Grafana dashboards
• Integrates with LangChain, LlamaIndex, Haystack, DSPy, CrewAI, Chroma, Qdrant, Weaviate, and LanceDB

Tasks

One SIE cluster runs the inference behind a whole agent. Each task is a handful of swappable models; every name below links to its config, whose sie_id is the model name you pass to the SDK (hf_id is the Hugging Face repository the weights load from; the two usually match). Browse packages/sie_server/models/ for the full set.

Task	What it does	Models
Search	Embed, match, and rerank to retrieve the right context.	bge-m3, splade-v3, colbertv2, qwen3-reranker
Document to markdown	PDFs, Office files, and scans become clean markdown.	lightonocr-3, lightonocr, glm-ocr, mineru, paddleocr-vl, docling
Structured output	Schema-valid JSON, extracted or generated.	gliner2, gliner-relex, gliformer, nuner-zero, qwen3.8-27b, qwen3.6-27b
Decide	Choice, yes/no, and score answers with probabilities to typed questions about a text or JSON state.	laya, laya-multilingual, laya-typed-decisions, gliner2.5-decide, gliner2.5-multi-decide, gliner2.5-decide-1b
Classify	Zero-shot labels, with several label groups answered in one call. The instruct models also follow a task instruction and few-shot examples.	gliclass-large-v3, gliclass-instruct-large, gliclass-multilang-mini
Guard content	A safety verdict: Safe, Unsafe, or Controversial with the unsafe categories; Yes/No with the threshold set in the model config; or safe/unsafe and policy-label scores with the threshold chosen per request.	qwen3guard-gen-4b, granite-guardian-2b, opir-multitask-large, opir-edge
Run the agent loop	Plan steps and call tools with an open LLM, streaming included.	qwen3.8-27b, qwen3.6-27b
Translate	Text between 400+ languages.	madlad400-3b-mt
See images	Caption, detect objects, and answer questions about images.	florence-2, owlv2, grounding-dino
Transcribe audio	Speech to text.	whisper-large-v3-turbo

Quickstart

The quickstart uses the smallest model in each family so the first run is fast on a laptop. Swap in any model from the table above for production quality.

1. Start the server

# macOS (Apple Silicon) or Linux, native (requires Python 3.12)
pip install "sie-server[local]" && sie-server serve

# Linux, NVIDIA GPU
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-default

# Linux, NVIDIA GPU: Transformers 5 models (`:transformers` OCR profiles, GLiNER2.5-Decide, and TopK-Embed-V1)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-transformers5

# Linux, NVIDIA GPU — SGLang vision OCR (default profiles of LightOnOCR-3, LightOnOCR-2, GLM-OCR, PaddleOCR-VL)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-sglang-vision-extract

# Linux, CPU
docker run -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cpu-default

Docker images are bundle-specific so dependency-incompatible model families stay isolated. Use the sglang-vision-extract image for LightOnOCR-3, LightOnOCR-2, GLM-OCR, and PaddleOCR-VL, or the transformers5 image for the :transformers profiles of LightOnOCR-3, LightOnOCR-2, and GLM-OCR, the GLiNER2.5-Decide models, and the TopK-Embed-V1 models; the default image intentionally does not advertise them.

# in a second terminal
curl http://localhost:8080/readyz   # expect: ok

The server speaks the OpenAI API out of the box, embeddings and generation alike (the cluster gateway serves /v1/chat/completions, /v1/completions, and /v1/responses). Your first call needs nothing but curl:

curl http://localhost:8080/v1/embeddings \
  -H 'Content-Type: application/json' \
  -d '{"model": "sentence-transformers/all-MiniLM-L6-v2", "input": "Hello world"}'
# {"object": "list", "data": [{"object": "embedding", "embedding": [-0.0344, 0.0310, ...

Each model's first call downloads its weights (progress appears in the server terminal). Later calls skip the download; inference latency depends on the model, task, hardware, and batch size.

2. Install the SDK

pip install sie-sdk                # Python
npm install @superlinked/sie-sdk   # TypeScript (pnpm and yarn work too)

3. Generate embeddings, rerank, and extract entities

from sie_sdk import SIEClient
from sie_sdk.types import Item

client = SIEClient("http://localhost:8080")

# Generate embeddings
result = client.encode("sentence-transformers/all-MiniLM-L6-v2", Item(text="Hello world"))
print(result["dense"].shape)  # (384,)

# Rerank search results
scores = client.score(
    "cross-encoder/ms-marco-MiniLM-L-6-v2",
    Item(text="What is machine learning?"),
    [Item(text="ML learns from data."), Item(text="The weather is sunny.")],
)
print(scores["scores"][0])  # {'item_id': 'item-0', 'score': -7.1, 'rank': 0}

# Extract entities
result = client.extract(
    "urchade/gliner_multi-v2.1",
    Item(text="Tim Cook is the CEO of Apple."),
    labels=["person", "organization"],
)
print(result["entities"][0])
# {'text': 'Tim Cook', 'label': 'person', 'score': 0.992, 'start': 0, 'end': 8, ...}

# Entities and a schema-valid record from one forward pass
result = client.extract(
    "knowledgator/gliformer-base-v1",
    Item(text="Refund request from Maria Lopez for order 4471 was denied: the item was returned after 45 days."),
    labels=["person"],
    output_schema={
        "type": "object",
        "properties": {
            "order number": {"type": "string"},
            "reason": {"type": "string"},
            "decision": {"type": "string", "enum": ["approved", "denied", "escalated"]},
        },
    },
)
print(result["data"])
# e.g. {'order number': '4471', 'reason': 'the item was returned after 45 days', 'decision': 'denied'}

Typed decisions ask one or more typed questions about each item (a text, or a JSON object or conversation passed as metadata={"state": ...}) and return an answer with probabilities per question in data. laya and laya-typed-decisions calibrate the probabilities with their shipped temperatures; laya-multilingual ships none, so its probabilities are the raw softmax. usage.input_tokens counts every (item, question) row the model encodes: the state's tokens plus the question's, once per question. An item may use up to 32,768 row tokens (questions × max_len), so the 1024-token laya-multilingual and laya-typed-decisions take up to 32 questions per request and laya up to 64.

result = client.extract(
    "convaiinnovations/laya",
    Item(text="Hi, we were billed twice for March. Please refund the duplicate today."),
    output_schema={
        "department": {"type": "choice", "instructions": "Which team should handle this?",
                       "criteria": {"billing": "invoices, payments, refunds", "technical": "bugs, outages"}},
        "refund_requested": {"type": "noul", "instructions": "Does the user ask for a refund?"},
        "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["low", "medium", "high"]},
    },
)
print(result["data"]["department"])  # values are illustrative and rounded
# {'type': 'choice', 'choice': 'billing', 'probabilities': {'billing': 0.987, 'technical': 0.013}, 'confidence': 0.9}

The GLiNER2.5-Decide models (fastino/GLiNER2.5-Decide for English, GLiNER2.5-multi-Decide, and GLiNER2.5-Decide-1B), served by the transformers5 image from step 1, take the same questions, GLiClass-style options={"label_groups": {...}}, or plain labels, and return every option's probability. They read all of a call's questions next to the document in one row per item: one forward pass answers them all, and each question's probabilities depend on the other questions sent with it. A score question is read as the ordinal labels "0" to "k-1", each described by its criterion; a noul question as "yes"/"no". usage.input_tokens counts the document tokens the model reads plus the questions' instructions and criteria text; question ids and label names are not counted.

result = client.extract(
    "fastino/GLiNER2.5-Decide",
    Item(text="Guest in room 1408 says the AC has been out since yesterday and wants to move rooms tonight."),
    output_schema={
        "intent": {"type": "choice", "instructions": "What does the guest want?",
                   "criteria": {"room_change": "move to another room", "maintenance": "fix something", "checkout": None}},
        "needs_human": {"type": "noul", "instructions": "Must a person act on this?"},
        "urgency": {"type": "score", "instructions": "How urgent is this?", "criteria": ["low", "normal", "high", "urgent"]},
    },
)
print(result["data"]["intent"])  # values are illustrative and rounded
# {'type': 'choice', 'choice': 'room_change',
#  'probabilities': {'room_change': 0.871, 'maintenance': 0.085, 'checkout': 0.045}, 'confidence': 0.57}

Text generation runs on the GPU generation image; stop the first server, then start this one on the same port:

# Linux, NVIDIA GPU (for generation on Apple Silicon via MLX, see the docs below)
docker run --gpus all -p 8080:8080 \
  -v sie-hf-cache:/app/.cache/huggingface \
  ghcr.io/superlinked/sie-server:latest-cuda12-sglang

result = client.generate(
    "Qwen/Qwen3-0.6B",
    "Reply with a single word: the capital of France.",
    max_new_tokens=16,
    temperature=0.0,
)
print(result["text"])  # Paris

For generation on Apple Silicon (MLX), the TypeScript walkthrough, and every configuration in between, see the quickstart guide, TypeScript SDK docs, and SDK reference.

---

Production

The same code works against a production cluster. SIE ships a load-balancing gateway and Kubernetes deployment surface, including Helm charts, KEDA autoscaling (scale to zero), and Grafana dashboards. Public Terraform modules are maintained separately for Alibaba Cloud ACK, EKS, AKS, and GKE. Not just the server, the whole stack. All Apache 2.0.

# pick one values overlay: values-ack.yaml / values-aws.yaml / values-aks.yaml / values-gke.yaml
# (pin a chart version for reproducible installs, e.g. --version 0.7.3)
helm upgrade --install sie-cluster oci://ghcr.io/superlinked/charts/sie-cluster \
  --namespace sie --create-namespace \
  --set hfToken.create=true \
  --set hfToken.value=YOUR_HF_TOKEN \
  -f https://raw.githubusercontent.com/superlinked/sie/main/deploy/helm/sie-cluster/values-gke.yaml

See the deployment guide.

Telemetry

sie-server sends anonymous usage telemetry so the maintainers can see which versions, platforms, and GPUs are in use. It is on by default in releases before 0.13.0. From 0.13.0 it is off unless SIE_TELEMETRY_ENABLED=1 is set. Until then, a server that sends it only because of the default logs a notice at startup.

Each event is a JSON POST to https://telemetry.superlinked.com/api/telemetry (SIE_TELEMETRY_URL overrides the endpoint) with these fields:

Field	Value
worker_id	A random UUID generated on first start and stored in the server data directory (SIE_DATA_DIR)
sent_at	The send time (UTC)
event	init, update, or terminate
sie_version	The installed sie-server version
variant	SIE_VARIANT: the image platform and bundle, set by the Helm chart (null otherwise)
os, arch	Operating system and CPU architecture
gpus	GPU model names
deployment_env	SIE_DEPLOYMENT_ENV (default unknown)

The receiver derives country, region, and city from the request's source IP address and does not store the address. No request data, hostnames, or file paths are sent.

init is sent at start, the first update 45 to 75 minutes later, then one update every 55 to 65 minutes, and terminate at shutdown.

• Opt in: SIE_TELEMETRY_ENABLED=1.
• Opt out: SIE_TELEMETRY_DISABLED=1 or DO_NOT_TRACK=1, which always win, or SIE_TELEMETRY_ENABLED=0.
• SIE_TELEMETRY_ENABLED and SIE_TELEMETRY_DISABLED accept 1, true, yes, on and 0, false, no, off, in any case. Any other value of either turns telemetry off with a warning.

Every start logs the state and where it came from, for example telemetry: on (default) or telemetry: off (DO_NOT_TRACK). The Helm chart's telemetry.enabled default also changes to false in 0.13.0; see the chart README. Policy: superlinked.com/policies/telemetry.

---

Explore

Model catalog: every model is a config in packages/sie_server/models/; pass its sie_id to the SDK.

Integrations: setup guides for all nine framework and vector-store integrations, in Python and TypeScript.

Examples: An end-to-end project gallery.

MCP edge: offload document, image, and structured-output work from Claude and other MCP clients to your cluster and save agent tokens.

Why we built SIE: The motivation, told at AI Engineer Europe 2026.

---

Development

Read CONTRIBUTING.md for the complete development workflow and AGENTS.md for repository automation boundaries.

Install mise, then bootstrap the versioned Python, Rust, Node.js, and Helm toolchains from the repository root:

./tools/init.sh

The common development checks and local server are available as mise tasks (mise tasks lists them all):

mise run test
mise run lint
mise run typecheck
mise run serve
mise run rust-check
mise run rust-test
mise run gateway-test
mise run server-sidecar-test
mise run helm -- dependencies
mise run helm -- lint --set payloadStore.enabled=false
mise run helm -- template --set payloadStore.enabled=false

The Python workspace uses the committed root lock. Package membership is explicit in pyproject.toml; a package joins the workspace only in the same change that adds its complete source. Native audio is always an opt-in build: install cmake, then run mise exec -- uv sync --frozen --project . --all-packages --all-extras.

---