Most engineers who run open-source LLMs in production have some version of the same benchmarking story. At some point, you wrote a Python script to compare your inference providers. It measured TTFT and tokens/sec, maybe it computed cost, and it gave you a winner. You updated your routing config based on the result, closed the terminal, and moved on. Six weeks later, a provider you'd deprioritized came back with improved hardware, or the provider you chose started showing latency spikes during peak hours, and you had no idea because the script was sitting in a local folder gathering dust.

One-off benchmarks answer a question at a single point in time. Production environments need answers that stay current.

What automated benchmarking actually needs to do

A useful automated benchmark setup does four things that a script run by hand cannot.

First, it runs on a schedule without someone triggering it. Inference provider performance changes day to day, sometimes hour to hour. A benchmark that only runs when you remember to run it will always lag behind reality.

Second, it stores history in a structured way. Knowing that Fireworks was fastest this morning is useful. Knowing that Fireworks has been fastest for 18 of the last 24 hourly runs, while Groq leads on 6, is more useful. Trends tell you whether a performance shift is real or a one-off anomaly.

Third, it alerts you when something meaningful changes. You should not have to check a dashboard to find out that your current provider's p95 latency has drifted 200ms above where it was last week. The system should tell you.

Fourth, it connects insight to action. A benchmark that tells you Provider B is now cheaper than Provider A is only useful if you can act on that quickly. Ideally, your tooling generates the config change for you.

Setting this up without building it yourself

Building a system like this from scratch is possible but genuinely tedious. You need to handle streaming correctly to measure TTFT accurately, run provider calls in parallel with per-call timeouts so one slow provider doesn't block the whole run, persist results in a queryable format, build trend logic on top of that data, add an alerting layer with debounce logic to prevent spam, and keep all of this running reliably on a cron schedule. Each piece is tractable on its own; together they're probably a week of engineering time to build and an ongoing maintenance burden to keep working.

Tesseract is built to replace that work. It handles the benchmarking infrastructure so you can spend your time on the application layer.

How to set up automated benchmarking in Tesseract

The setup takes about ten minutes.

Step 1: Connect your providers. Go to the Providers page and add your API key for each inference provider you use. Tesseract supports Fireworks, Together AI, Groq, Baseten, and OpenRouter. Each key is verified immediately with a lightweight models-list call to confirm it works, then stored encrypted. The free tier allows two connected providers; Pro removes that limit.

Step 2: Run a manual benchmark to establish a baseline. Go to the Benchmark page, select your model (Llama 3.1 70B, Mistral 7B, or whichever model you're routing in production), enter a representative prompt from your actual use case, and run it. This is important: your benchmark prompt should reflect what your app actually sends, not a generic test string. A short factual query and a long multi-turn reasoning prompt will produce different provider rankings, and you want the ranking that matters for your workload. Tesseract fires the same request to every connected provider in parallel, measures TTFT, tokens/sec, total latency, and cost per million tokens, and returns a ranked table. The run is saved and reachable at a permanent URL.

Step 3: Set up a scheduled benchmark. This is the step that turns a one-time test into ongoing monitoring. In the Benchmark section, you can schedule a benchmark for your model at hourly, 6-hour, or daily intervals. Tesseract runs the benchmark automatically on that cadence and updates your scoreboard with the results. The scoreboard on your dashboard shows a row per model with the current fastest and cheapest provider over the last 24 hours and 7 days, along with trend arrows. You'll see at a glance whether your current provider choice is still the best one. Scheduled benchmarks are a Pro feature.

Step 4: Create routing rules. Go to the Routing page and create a rule for your model with an objective: lowest latency, highest throughput, or lowest cost. Tesseract reads the current scoreboard data and returns the best provider under that objective, plus a ready-to-paste config with the correct base URL, model identifier, and a curl or Python snippet. You can copy this directly into your app. If the top provider has no recent data, the rule falls back to the next-best option and labels it clearly. The free tier supports one active routing rule; Pro removes that limit.

Step 5: Set latency and cost alerts. Go to the Alerts page and create an alert tied to your model with a threshold condition. You can alert on latency exceeding a millisecond threshold, cost exceeding a per-token threshold, or simply when a better provider appears for your configured objective. When a scheduled benchmark triggers the condition, you receive an email within about five minutes naming the model, the condition, and the better provider. Alerts debounce correctly: once triggered, they stay suppressed until the condition clears and re-triggers, so you get one notification per event rather than a flood. Alerts require Pro.

What good monitoring looks like in practice

Once this is running, your day-to-day interaction with inference provider performance changes significantly. Instead of manually checking whether your provider is still competitive, you have a scoreboard that reflects the last automated run whenever you open it. Instead of finding out about a latency regression from a user complaint, you get an email when it happens. Instead of manually translating benchmark results into a config change, you paste the code snippet Tesseract generates.

The engineers who get the most value from this setup are the ones with two or more providers connected and an automated benchmark running at least every six hours. That cadence catches most performance shifts before they affect users in any meaningful way. Hourly is better for latency-sensitive applications where a 200ms regression matters.

The free tier is enough to start

You can connect two providers and run five manual benchmarks per day on the free tier at tesseract.click. That's enough to run a real comparison against your actual keys and see whether the provider you're using today is still the best option for your model. Scheduled benchmarks, unlimited providers, alerts, and routing rules are all on the Pro tier at $39 per month.

If you're currently maintaining a benchmark script, the question worth asking is how long ago you last ran it.