hunch: millisecond-level AI decisions on cheap CPUs

hunch is an open-source, local semantic decision system built on Google's EmbeddingGemma 2, designed to execute millisecond-level judgments directly on commodity CPUs.

In modern agent loops, a massive amount of token spend and latency comes from trivial, high-frequency verifications. Developers constantly query frontier models just to check if a tool call failed, inspect streaming logs for anomalies, detect prompt injections, or route user intents. hunch is engineered to offload these repetitive questions.

How it learns on the fly

hunch functions as a fast System One reflex engine using a teacher-student pattern. It combines EmbeddingGemma 2, a lightweight classifier, and a continuous distillation loop.

Your existing judge or frontier LLM acts as the teacher, while hunch's local classifier serves as the student. The student evaluates semantic vector embeddings generated by EmbeddingGemma 2.

Initially, all queries route to the teacher model while hunch trains in the background. Once cross-validation proves that the student agrees with the teacher frequently enough, the model graduates and begins answering requests locally on its own.

After graduation, hunch stays in check through continuous auditing. It periodically passes samples back to the teacher to verify accuracy, and if an input lands too far outside its training domain, it automatically punts the question back to the teacher rather than guessing.

Hardware specs and benchmark numbers

hunch requires zero GPU infrastructure. Its core relies on eg2cpu, an optimized engine written in Rust tailored for commodity CPUs.

On an Apple M5 Pro chip, hunch delivers a p50 text decision latency of 22 milliseconds on a 190-token tool trace. Image evaluations process in roughly 100 milliseconds. Operational costs drop below 3 cents per million tokens.

Under load, it sustains 86 requests per second across 32 concurrent callers with a p99 latency of 466 milliseconds. When run on a 24-core Sapphire Rapids server utilizing AMX instructions, encoding 32 tokens down to layer 8 takes just 0.6 milliseconds per text in batches.

The project also ships with hunch.live, a library available in Python and TypeScript that allows developers to set up real-time decision flows over text and camera feeds.

Getting started and limitations

hunch, the eg2cpu engine, and related examples are published under a dual MIT and Apache-2.0 license on GitHub at undrgrnd-labs/hunch. The underlying EmbeddingGemma 2 checkpoint carries an Apache-2.0 license.

Running it locally requires Rust 1.88 or later along with the uv package manager. You can compile the extension with maturin and retrieve model weights directly using the provided download script.

The tool is designed strictly for structured judgments like binary checks, multi-choice classification, or scale ratings. It does not perform deep deductive reasoning, meaning open-ended diagnostics still belong with your primary LLM.

Why it matters

Routing every micro-decision back to a frontier cloud model is becoming an unsustainable bottleneck for complex agent pipelines. hunch demonstrates a clean tiered-reasoning approach that treats fast local embeddings as a frontline buffer.

For developers building high-throughput agent workflows, this is an immediate way to slash inference bills and cut trace latency without buying expensive accelerators. It is practical, lightweight, and well worth testing on routine trace-monitoring workloads.