Project brief
From scenario and baseline to reproducible evidence.
Build an AI gateway that selects models by task, budget, latency, and risk, then improves through independently evaluated, reversible updates.
The handbook starts with a runnable minimum, then adds real constraints. The final presentation should explain the method, results, failures, limitations, and next steps.
Evaluation focus
Use metrics to make trade-offs visible.
Project handbook
Project brief
Create a gateway for a mixed task set such as code repair, information extraction, structured question answering, or summarization. Compare at least three model tiers or controlled strategies against a static routing baseline.
Minimum version
Start with a unified interface that records task features, model choice, output, token usage, latency, cost, and scores.
- A static routing baseline and at least three model tiers or mock services.
- A trace and failure taxonomy covering quality, timeout, tool, format, and budget failures.
- An RSL loop that proposes two to five structured routing candidates.
- An independent validation set and an immutable acceptance threshold.
Evaluation and boundaries
Report quality, cost, and tail latency together. The system must not change its evaluation rules, delete logs, elevate tool permissions, or call real production systems. Every accepted policy needs a version, diff, and rollback path.
Resource note: External code, models, datasets, and platforms remain subject to their own licenses, terms of service, and applicable law.