Together AI Releases Together Compiler: Agents Make Inference Up to 2.6x Faster

Together AI just dropped its Together Compiler (TKCC), a self‑tuning inference engine that promises to keep your LLM serving stack from going stale. The claim is simple: never stop tuning, because traffic never sits still. As someone who’s been wiring up production servers for a decade, I’m skeptical of any hype that sounds like “set it and forget it.” But the numbers they’ve released—up to 2.6× higher TPS on a handful of models, 252× more agent experiments than a restart‑based baseline—force a closer look. Below I break down what the system actually does, where the real gains come from, and what this means for independent hosting providers who can’t afford to chase every new optimizer that pops up.

Continuous compilation, not one‑off tuning

The core of TKCC is a loop that runs forever. You feed it a launch command, a traffic profile and a hardware spec (they list NVIDIA B200, B300, GB300, H100, AMD MI355X and AWS Trainium2). Coding agents then rewrite the model’s forward pass, submit the candidate over RPC, and a persistent engine measures the result. The best candidate is hot‑reloaded into the serving engine without a full restart.

This matters because the team observed that prompt lengths can vary dramatically across geographies—one country saw 97 K tokens versus 70 K elsewhere. One customer’s P90 input length jumped from 18 K to 33 K tokens in just three days. Those shifts change memory pressure and kernel behavior, so a static tuning that was optimal yesterday can become a bottleneck today. By keeping the compilation loop alive, TKCC can react to that drift automatically.

Hot‑reload architecture beats restart cycles

Typical agent loops in the research world restart the entire inference stack (vLLM, SGLang, TensorRT‑LLM) for each experiment. Together measured a 761‑second cold start for MiniMax M3.1 Flash on TensorRT‑LLM. TKCC instead starts the engine once, swaps the forward function pointer, and re‑captures CUDA graphs. That hot‑reload approach yields a reported 252× higher agent experiment throughput compared with the restart baseline.

They also introduced a two‑tier measurement system: a fast tier that runs 12 layers for quick screening, and a slow tier that runs 60 layers for fidelity. The two‑tier setup tested 1,354 candidates versus 168 with a single slow tier—a boost of 8.1× in candidate coverage. The extra screening found a solution 7.2% faster than the best single‑tier result, showing that the extra engineering effort translates into real performance gains.

Real‑world gains versus lab‑only numbers

The headline claim is up to 2.6× higher TPS over the best InferenceX configurations, measured on a P90 workload with batch size 1. The best‑performing case was MiniMax M3.1 Flash on an NVIDIA B200 GPU, delivering 1,111 TPS versus a baseline of 633 TPS—a 1.8× uplift. Other models showed more modest improvements: DeepSeek v4.1 Flash on a GB300 hit 1,205 TPS versus 943 TPS; GLM 5.3 on an MI355X moved from 463 to 480 TPS (about a 4% gain); and on Kimi K3 the gain was 450 versus 423 TPS.

Beyond LLMs, TKCC sped up eight scientific models by 1.5× to 6.4× with bit‑exact outputs, and on Llama 1B it beat Hazy Research megakernels, Mirage and CUDA‑graph baselines by up to 1.7×. Those numbers are self‑reported, but they cover a range of hardware and model families, suggesting the approach isn’t limited to a single niche.

Cost model and parallelism layout selection

Before the agents start rewriting code, TKCC runs a cost model that ranks tensor, expert, data and context parallel layouts. It blends spec‑sheet numbers, micro‑benchmarks and measured forward passes, also factoring in MoE expert imbalance and uneven request lengths. On the Kimi K3 model, the cost model identified a layout that delivered up to 24% higher TPS in production. This pre‑selection step steers the agents toward promising regions of the search space, avoiding wasted cycles on clearly sub‑optimal configurations.

For MiniMax, the cost model chose TP8 (tensor parallelism of 8) over the baseline TP4, effectively halving each GPU’s memory‑bound decode traffic. That change alone contributed significantly to the 1.8× TPS lift, showing that layout decisions can be as impactful as low‑level kernel tweaks.

Agent behavior: naive beats expert knowledge

One surprising finding is that a curated knowledge base actually hampered speedup. An expert‑curated knowledge base produced 25% to 31% lower speedup after 12 hours of tuning. Naïve agents instructed simply to “make the forward pass as fast as possible” outperformed expert‑assigned strategies, achieving ensemble scores of 6.2× versus 4.9× after ten trials. Custom logging structures also ate up to 43% of tool calls, further eroding efficiency.

The takeaway for us in the hosting world is that over‑engineering the search with heavy human‑crafted heuristics can backfire. Let the agents explore broadly, and keep the measurement loop lean. The raw engine source code—rather than an intermediate representation—gave agents the freedom to rewrite kernels directly, yielding 1.7× to 2.5× speedups on specific kernels like Kimi K3’s MLA metadata kernel.

Practical implications for independent providers

For a boutique hosting outfit, the biggest win from TKCC is the ability to adapt to traffic drift without a full redeploy. The hot‑reload mechanism means you can push a new forward pass in seconds, not minutes, and you avoid the 761‑second cold start penalty that can cripple SLA commitments during a rollout. The two‑tier measurement system also keeps experimentation cheap: the fast tier runs a fraction of the model, letting you screen thousands of candidates before committing to the slower, high‑fidelity validation.

However, the system still relies on up to two nodes per model for each 24‑hour tuning run. Small providers will need to allocate dedicated hardware for the tuning loop, separate from production serving, or risk contaminating latency. The cost model’s reliance on spec‑sheet numbers and micro‑benchmarks means you’ll need to maintain an up‑to‑date hardware inventory to get accurate layout suggestions.

Actionable steps to get started

1. **Instrument your traffic** – Capture prompt length distributions and request patterns per region. The TKCC paper shows that a shift from 18 K to 33 K tokens in three days can materially affect performance.

2. **Set up a persistent engine** – Deploy a long‑running inference server (vLLM, SGLang or TensorRT‑LLM) and expose an RPC endpoint for agents to submit forward‑pass candidates. Avoid full restarts; instead swap the forward function pointer and re‑capture CUDA graphs.

3. **Adopt a two‑tier validation** – Implement a fast screening tier (e.g., 12‑layer snapshot) and a slow confirmation tier (full 60‑layer run). This will let you evaluate an order of magnitude more candidates without blowing up measurement time.

4. **Leverage a simple cost model** – Use spec‑sheet FLOP counts, memory bandwidth, and measured MoE expert activation rates to rank parallelism layouts before agents start. Even a coarse model can surface a 24% TPS lift as seen on Kimi K3.

5. **Keep the agent loop lean** – Skip heavy expert knowledge bases and custom logging that consume large fractions of tool calls. Let agents work directly on raw source code for the biggest kernel wins.

6. **Plan hardware budgeting** – Allocate at most two nodes per model for the 24‑hour tuning window. For smaller setups, consider rotating models through the tuning loop rather than running all concurrently.

7. **Monitor for drift** – Set alerts on changes in average or P90 prompt length, and trigger a new TKCC run when thresholds are crossed. Continuous re‑optimization is only valuable if you actually detect the drift.

In short, Together Compiler is not a magic bullet, but it does deliver a pragmatic path to keep inference stacks humming as traffic evolves. For independent hosting providers, the combination of hot‑reload, two‑tier screening and a lightweight cost model offers a cost‑effective way to squeeze out up to 2.6× more throughput without buying into the endless hype of hyperscaler‑only solutions. Keep your servers lean, keep your agents busy, and never let a static tuning become your Achilles heel.

— Allan Ali, Founder

This article was produced with AI-assisted research and editorial support. Reporting is based on the source material cited below. Sources: MarkTechPost; marktechpost.com; Global1.News (05 November 2026).

By Allan Ali, Global1.News