Milestone 2.5.G4.M2 — TEI Backend (GPU Profile)

Status: Completed (2026-08-24)
Goal: Track D — Pluggable Embedding Service
Phase: 2.5 — Provider Generalization & Foundation
Estimated effort: 1–2 days

Why This Milestone Exists

Implement the production/GPU embedding backend. TEI ships as a prebuilt Docker image (ghcr.io/huggingface/text-embeddings-inference) with a model ID passed as a launch arg — this milestone is a thin HTTP client, not custom inference code. tei_backend.py calls TEI (POST /embed, with OpenAI-compatible /v1/embeddings also supported) and stays smaller than hand-rolled batching-buffer-timeout logic because TEI's Rust core already implements continuous batching, dynamic padding, and flash-attention kernels.

TEI is Apache-2.0, fully free, maintained by Hugging Face — no licensing concern, no vendor lock-in (it's a standard HTTP server can later be swapped for vLLM's embeddings endpoint or Infinity later without changing EmbeddingBackend's interface).

Non-Goals

Custom batching logic (TEI handles this internally)
GPU provisioning automation
Inference performance tuning beyond TEI's defaults

Orientation (indicative)

Named paths, package layouts, libraries, schemas, env vars, and commands anywhere on this page are rough sketches for orientation — inspiration and a baseline, not a required change list.

During implementation, expect to:

open the live tree and follow existing patterns before inventing new ones
research current constraints (latency, tenancy, deploy shape, libraries) more deeply than this page can
advance the design beyond the sketch where measurement or code reality says so
land work in different filenames, merged packages, deferred docs, or new surfaces when the situation calls for it

Prefer outcomes over matching any particular file tree or command sequence.

Areas that may be involved (situational — not a checklist):

Embedding service / backends

Suggested naming (provisional)

Rename freely to match the change that actually lands.

Branch: feature/m2-5-g4-m2-tei-backend
PR title: feat(embedder): TEI HTTP backend for GPU/production embedding profile (m2.5.G4.M2)

Deployment Configuration

YAML

# docker-compose / k8s equivalent
embedder-tei:
 image: ghcr.io/huggingface/text-embeddings-inference:1.6
 command: ["--model-id", "BAAI/bge-m3", "--port", "80"]
 # GPU variant requires nvidia container runtime
 deploy:
 resources:
 reservations:
 devices:
 - driver: nvidia
 count: 1
 capabilities: [gpu]

Working notes

Preferred starting points and open questions — situational, and expected to evolve with further research during implementation.

TEI backend is a thin HTTP client

Python

# Illustrative — exact path may differ
import httpx
import numpy as np
from .base import EmbeddingBackend
 
class TEIBackend(EmbeddingBackend):
 def __init__(self, base_url: str, model_id: str = "BAAI/bge-m3"):
 self._client = httpx.AsyncClient(base_url=base_url, timeout=30.0)
 self._model_id = model_id
 self._dimensions = 1024 # bge-m3 output dim
 
 async def embed(self, texts: list[str]) -> np.ndarray:
 resp = await self._client.post("/embed", json={"inputs": texts})
 resp.raise_for_status()
 return np.array(resp.json(), dtype=np.float32)
 
 @property
 def dimensions(self) -> int:
 return self._dimensions
 
 @property
 def name(self) -> str:
 return "tei"

Contract test

The contract test suite (defined in 2.5.G4.M1) should pass:

Same input → vectors of shape (n, 1024) (bge-m3 dimensionality)
Vectors are L2-normalized (norm ≈ 1.0)
Batch and single-item inputs produce identical per-item vectors

Success signals

Prerequisites

2.5.G4.M1 (Embedder interface and registry) — EmbeddingBackend ABC and contract test suite