We're Gabriel & Viraj, and we're excited to open source TensorZero.
To be a little cheeky, TensorZero is an open-source platform that helps LLM applications graduate from API wrappers into defensible AI products.
1. Integrate our model gateway
2. Send metrics or feedback
3. Unlock compounding improvements in quality, cost, and latency
It enables a data & learning flywheel for LLMs by unifying:
• Inference: one API for all LLMs, with <1ms P99 overhead
• Observability: inference & feedback → your database
• Optimization: better prompts, models, inference strategies
• Experimentation: built-in A/B testing, routing, fallbacks
Our goal is to help engineers build, manage, and optimize the next generation of LLM applications: AI systems that learn from real-world experience.
In addition to a Quick Start (5min) [1] and a Tutorial (30min) [2], we've also published a series of complete runnable examples illustrating TensorZero's data & learning flywheel.
• Writing Haikus to Satisfy a Judge with Hidden Preferences [3] – my personal favorite
• Fine-Tuning TensorZero JSON Functions for Named Entity Recognition (CoNLL++) [4]
• Automated Prompt Engineering for Math Reasoning (GSM8K) with a Custom Recipe (DSPy) [5]
___
[1] https://www.tensorzero.com/docs/gateway/quickstart
[2] https://www.tensorzero.com/docs/gateway/tutorial
[3] https://github.com/tensorzero/tensorzero/tree/main/examples/...
[4] https://github.com/tensorzero/tensorzero/tree/main/examples/...
[5] https://github.com/tensorzero/tensorzero/tree/main/examples/...
We hope you find TensorZero useful! Feedback and questions are very welcome. If you're interested in using it at work, we'd be happy to set up a Slack channel with your team (free).