Two local models. One endpoint.
tinygw picks which one answers.
tinygw is a small OpenAI-compatible gateway that runs next to your model servers. For each request it decides: strong model or light model. It forwards the call, streams the answer back, and writes nothing down.
client ──► tinygw :3000 ─┬─► strong Flash-Next :8000
└─► light Inkling 2 Mini :8001
- Zero logs no prompts or completions on disk, stdout or in a logger
- Local binds to 127.0.0.1, nothing leaves the machine
- Small ~1k lines of Python, FastAPI + httpx
- Drop-in
/v1/chat/completions, streaming included
How a request is routed
Checked in this order. The first step that decides wins.
- 1
Forced
Header
x-tinygw-route: strong, modeltinygw/strong, or the exact upstream model id. You always have the last word. - 2
Hard rules
Code fences, diffs and patches, words like audit, review, security, or a prompt over ~6,000 tokens go to the strong model. A message under 80 characters goes to the light one. No model is called.
- 3
Decision model (optional)
Only what the rules left open. A small decision model such as Liquid AI's d1-omni-600M (MLX, Apple Silicon) answers one question, strong or light, with a calibrated probability. One forward pass, no generated text.
- 4
Confidence fallback
If that probability is under 0.8, the request goes to the strong model. When unsure, spend compute instead of mis-routing.
Every response says what happened: x-tinygw-route, x-tinygw-stage, x-tinygw-reason, x-tinygw-confidence.
Quickstart
# install
git clone https://github.com/jackjohns839/tinygw && cd tinygw
uv venv && uv pip install -e .
cp tinygw.example.toml tinygw.toml
# see where a prompt would go, and why
tinygw route "review this auth middleware for token leaks"
route=strong stage=rule reason=keyword:review
# run it, then point your client at http://127.0.0.1:3000/v1
tinygw serve
# optional: the d1-omni decision model on Apple Silicon
uv pip install -e '.[d1omni]'
TINYGW_BACKEND_NAME=d1omni tinygw serve
# tinygw.toml
[strong]
base_url = "http://127.0.0.1:8000/v1"
[light]
base_url = "http://127.0.0.1:8001/v1"
[backend]
name = "rules" # or "d1omni"
confidence_threshold = 0.8
Routing is hard. We say so.
Deciding which model should answer means guessing how hard a request is before anyone has answered it. Published evaluations show off-the-shelf decision models doing close to chance at exactly this. Small ones score modestly on general benchmarks too.
So tinygw doesn't put a model in charge. Rules handle the clear cases, the model only sees the fuzzy middle, and low confidence means the strong model. The default backend uses no model at all. Try a decision model on your own traffic with tinygw route before you trust it.
Why not a big gateway?
LiteLLM, Portkey and friends handle many providers, keys, budgets and dashboards. tinygw is for one machine with two local models: no database, no UI, nothing persisted, three dependencies. If you need provider fallbacks or spend tracking, use one of those.
Source
github.com/jackjohns839/tinygw · Apache-2.0 · The repository is private for now and will open up once it has had more real-world use.