Together AI, the AI-native cloud, today announced Together Compiler, which uses teams of AI agents to rewrite the GPU code behind the models it serves. New open source models now get faster overnight instead of in weeks, with no additional work from customers. Together Compiler is the first step towards recursive self-improving inference and serves trillions of tokens per day.
Today’s AI workloads are rapidly changing. New models are released often, model architectures are increasingly diverse, and traffic distributions change frequently. 80% of OpenRouter traffic goes to models released in the last three months, and each of the ten most popular open models on OpenRouter uses a different architecture. To address the dynamic nature of AI inference, Together Compiler leverages agents to continually optimize new deployments.
Agents have proven their ability to optimize production code, but validating these changes are costly. In traditional systems, a single agent’s edit can take up to 15 minutes to validate in an inference engine. Together Compiler’s novel design maximizes agent iteration velocity, running 252x more experiments than existing systems that use agents for inference optimization, keeping costly GPUs fully utilized. Higher agent throughput yields state-of-the-art optimizations in 24 hours and can be immediately deployed to production, where Together Compiler currently serves trillions of tokens.
“Running Together Compiler cut our median time to first speech by nearly 50%, and it happened overnight,” said Moin Nadeem, Co-founder, Phonic. “Getting Phonic-1 optimized that quickly let us ship sooner. We’ve been a customer of Together AI for over two years and are excited to keep building with them as they find new ways to push performance.”
In testing across popular open models, Together Compiler:
- Tuned in under a day, across models and chips: On Kimi K3 and GLM 5.3, Together Compiler reached 1.5x and 1.4x higher peak throughput than the best result on SemiAnalysis’s InferenceX, an open inference benchmark.
- AI-written optimizations beat code expertly tuned by hand, at every length tested: On Llama 1B, Together Compiler reached over 2,100 tokens per second and up to 1.7x the speed of hand-written megakernels, including Together AI’s own. It was fastest at every output length tested on both NVIDIA H100 and B200 GPUs.
- The fastest single-user speeds we’ve measured: MiniMax M3.1 Flash reached 1,111 tokens per second on NVIDIA B200 GPUs, 1.5x the baseline. DeepSeek v4.1 Flash reached 1,205 tokens per second on NVIDIA GB300, 1.2x the baseline.
“Most inference today uses a fraction of what the GPU can actually do, and that gap opens back up every time a new model or chip comes out,” said Dan Fu, Founding SVP of Research at Together AI. “Models are changing every few weeks, often introducing novel architectures. Closing that gap for good requires inference that adapts just as quickly and keeps pace over time. Together Compiler is the first step towards a self-improving inference stack.”
Together Compiler is the first project of a larger initiative Together AI calls Polymorphic Inference, a form of recursive self-improvement for inference workflows. As models, workloads, and chips change faster than ever, inference for those workloads needs to be tuned frequently. Polymorphic Inference is Together AI’s research agenda for inference that adapts on its own, with AI agents continuously improving the systems that run AI.
“When presented with a well-specified target, AI agents are effective at hill-climbing on target goals across mathematics, engineering, and science,” said Simran Arora, SVP of Research, AI Systems, at Together AI. “The key challenge for Together AI is that our traffic shape, i.e. our target to optimize, is constantly evolving. This requires continual inference compilation and optimization. We have learned a ton of lessons from building an agentic inference compiler that keeps up with the pace of traffic evolution, the ~polymorphism~, within Together’s inference service.”
Together Compiler is running today across many of Together AI’s most popular serverless endpoints. Customers can join the limited beta to use it on their own dedicated deployments, with self-serve access coming soon.
Learn more at: www.together.ai/polymorphic-inference. The Together Compiler research paper is available at: www.together.ai/together-compiler-paper.
About Together AI
Together AI is the AI Native Cloud, combining state-of-the-art open-source models, high-performance infrastructure, and frontier research in AI efficiency and scalability. Founded in 2022, Together AI powers over a million developers and some of the world’s most demanding AI workloads, delivering production-scale inference, training, and reinforcement learning for the next generation of AI-native companies.
View source version on businesswire.com: https://www.businesswire.com/news/home/20261104879153/en/
Media gallery
