Building Compass: An Open System One Model

A customer writes: “I was charged twice and the app crashed this morning. Please refund the duplicate payment.”

A support system has a bounded decision to make. Which team should receive it: billing, technical, or sales? A chat model can write an answer. Compass picks from the three choices the application supplied and returns a probability for each.

billing:   0.92
technical: 0.07
sales:     0.01

That is Compass: an open decision model built on Qwen3.5-4B. It returns Choice, Noul, and Score answers without generating an answer sentence.

A Choice picks one named option. Noul returns the probability of yes for a yes-or-no question. Score returns a distribution across ordered levels such as low, medium, and high.

The repository includes the model adapter, server, trainer, calibration code, tests, and evaluation scripts.

System One and System Two models

Jev and Compass are System One models. They make a bounded decision from a state and a list of answers chosen by the application. Their response is a probability distribution such as billing: 0.92, technical: 0.07, sales: 0.01. The product can route the high-confidence ticket or escalate the uncertain one.

Qwen with thinking enabled is a System Two model. It generates a written reasoning trace before it produces an answer. On a reporting-chain question, it can write Maya → Priya → Omar → Riya, count three links, then answer yes. Those generated tokens give it a scratchpad for a task that needs several steps.

System 1 and System 2 thinking describe the distinction in human decision-making. For software, the useful distinction is practical: System One makes a fixed, typed decision; System Two can spend more generation on a hard problem and leave intermediate work in its context.

A chat completion also generates text from an open-ended vocabulary. A System One API restricts the answer to the choices the application supplied and returns a score for each. That makes it useful when code, rather than a reader, must decide what happens next.

The connection to Jev

Jev introduced a public System One API for this kind of work: send a state and typed questions, receive probabilities over caller-defined answers. Compass implements that public task contract and the same /v1/systemone wire format, so an existing TypeSafe client can call Compass by changing its base URL.

Compass uses Qwen3.5-4B, a LoRA adapter, candidate scoring, and temperature calibration. It is an independent implementation of Jev’s public interface and interaction model.

The shared API matters because it gives developers one request shape for a decision model.

When an answer needs a working trace

The test asks a simple question over an organisation chart: is one employee exactly a fixed number of manager links above another? In the three-link example, a correct solver follows the requested reporting path, counts the links, then answers yes or no.

I generated 20 balanced yes-or-no questions at each depth: 1, 2, 3, 4, 6, 8, and 12 links. I sent the same 140 questions to Compass 0.2.0, Jev 1.13.0, and Qwen3.5-9B with thinking enabled. Compass and Jev returned a probability for the same typed question. Qwen wrote a reasoning trace, then returned FINAL: YES or FINAL: NO.

Qwen3.5-9B finished 129 of 140 questions correctly, or 92%. Compass got 82 of 140 (59%) and Jev got 91 of 140 (65%). Every Qwen response that reached a final answer was correct. The 11 misses had no final answer.

On this balanced manager-chain test, Compass and Jev sit at or near the 50% chance baseline from two links onward. They return the fixed yes-or-no score without generating a path. Qwen reaches 100% at 6, 8, and 12 links after it writes and inspects a path.

I ran Qwen through OpenRouter on Parasail’s bf16 serving path at temperature zero, with 12 concurrent requests. Ten of its 11 incomplete responses stopped after 301 to 316 seconds. The remaining one stopped after 917 seconds.

Each was a loop without a final answer, so I scored it as wrong. The loops appeared only at depths 1 through 4.

I served Compass locally on an RTX 4090 and called Jev serially from that same host. After five Compass warm-up requests, Compass finished at 287 ms median and 324 ms p95. Jev finished at 334 ms median and 399 ms p95. These are client-observed deployment routes: Compass runs on localhost; Jev crosses the TypeSafe API.

Qwen took 23.54 seconds at the median, 82× longer than Compass. It generated 444,748 answer tokens across the 140 questions; Compass generated none and Jev reported 2,800. The Jev API run cost $0.0028. The Qwen run cost $0.056. Compass uses a self-hosted GPU, so I do not put its hardware cost beside those API charges.

How Compass picks an answer

Take the support ticket. The application gives Compass the ticket, a routing question, and three candidates: billing, technical, and sales. Compass produces two distributions over those same three candidates, then combines them.

First, Compass checks each candidate on its own. For each check, the model sees the full ticket, the routing question, and the definitions of billing, technical, and sales. Compass then changes one line: the proposed answer.

• “Is billing the right team for this ticket?”
• “Is technical the right team for this ticket?”
• “Is sales the right team for this ticket?”

For this ticket, the model might return:

• Billing: yes 0.94, no 0.06
• Technical: yes 0.31, no 0.69
• Sales: yes 0.02, no 0.98

Compass divides each yes value by its matching no value, then scales the three results to total 100%.

• Billing: 0.94 ÷ 0.06 = 15.67
• Technical: 0.31 ÷ 0.69 = 0.45
• Sales: 0.02 ÷ 0.98 = 0.02

For billing, the three values total 16.14. The billing calculation is 15.67 ÷ 16.14 = 0.971. Compass performs the same division for technical and sales. That gives the candidate-check result: billing 97%, technical 3%, and sales less than 1%.

Second, Compass compares the candidates side by side. It puts the ticket, question, and the three team definitions into one prompt. The actual prompt has one temporary letter next to each team:

<document>
I was charged twice and the app crashed this morning.
</document>

Question: Which team should receive this ticket?
Allowed answers:
  A. billing: handles duplicate charges and refunds.
  B. sales: handles buying and account upgrades.
  C. technical: handles app faults and bugs.
Which one is the correct answer? Reply with its letter only.

A, B, and C are temporary index markers, not extra answers. Compass maps them straight back to billing, sales, and technical. The model still reads the real team names and definitions. The letters standardise the final answer token, so the comparison does not reward a team because its name appears more often in training text or breaks into different tokens. At the next response position, the model has a score for every possible token. Compass reads the scores for the three letter tokens only, scales those three scores to 100%, then maps the result back to the teams. If it gives A 80%, B 3%, and C 17%, the comparison result is billing 80%, sales 3%, and technical 17%.

Finally, Compass combines the two distributions. Treat the candidate check and the side-by-side comparison as two opinions about the same ticket. Compass gives them equal weight and rewards agreement. Billing stays high because both checks favour it. Technical stays lower because neither check puts it first.

• Candidate check: billing 97.1%, technical 2.8%, sales 0.1%.
• Side-by-side comparison: billing 80%, technical 17%, sales 3%.

Compass multiplies the two results for each team and takes the square root.

• Billing: √(0.971 × 0.80) = 0.881
• Technical: √(0.028 × 0.17) = 0.069
• Sales: √(0.001 × 0.03) = 0.005

Those three values total 0.956. Compass divides each by 0.956, giving billing 92%, technical 7%, and sales 1%. A final calibration step adjusts the confidence level using held-out examples.

The full flow is:

1. Compass reads the ticket and routing rubric.
2. It tests billing, technical, and sales one at a time, then turns the three yes/no results into a candidate-check result.
3. It shows the same choices together and creates a comparison result.
4. It multiplies the matching scores, takes the square root, then scales the result to 100%.
5. It calibrates the result and returns probabilities for billing, technical, and sales.

The steps above reuse the same ticket. Compass reads the ticket once, keeps the model’s internal working memory for it, called a KV cache, and reuses that memory for the short candidate checks. The cache improves speed and cost. It does not change the decision logic.

The application still owns the consequences. A high score can route a ticket. A low score can send the ticket to a person or a larger model. The model cannot invent an answer outside the answer set.

How I built Compass

I built Compass as a small decision stack: a contract compiler that validates the request, a Qwen scorer that rates the candidates, a fusion rule that combines the two readouts, and a calibration step that turns raw scores into useful probabilities. The server exposes that stack through the public System One API.

I trained Compass on examples with three parts: a state, a typed answer set, and the expected answer distribution.

For each example, the training code scores the candidates using the same two checks used by the server. When the scores favour the wrong answer, the LoRA adapter updates. That keeps training and serving aligned.

We also change the presentation of each decision during training and evaluation.

• We reorder the same answer options.
• We replace meaningful labels such as billing with neutral labels such as option_3.
• We include hard cases where two options look plausible.

Those checks help us catch a model that learned the ordering or label names instead of the actual criterion. We then fit a temperature on held-out examples so the returned probabilities better match observed accuracy.

You can build a smaller version without training a new model.

1. Pick one decision with a small answer set, such as “send this email” or “ask for approval.”
2. Collect examples that represent the cases your product must handle, including cases that should defer.
3. Score each candidate by asking whether it fits the state. Compare the scores.
4. Test renamed answers, reordered options, and misleading irrelevant text.
5. Add a product policy for low confidence before you automate an action.

The Compass code contains the contract compiler, scorer, trainer, calibration step, and release scripts.

JevBench

JevBench is a public benchmark for System One decision models: it tests probability answers to typed decision questions. We used its public tiers as a transfer check. The public JevBench files never enter Compass training, calibration, or release selection.

I opened JevBench issue #53 to submit Compass for the maintainer-run v1.4 evaluation.

Scope CheckChecking an agent’s action

Scope Check shows how a decision model can sit before an agent action.

A user asks an agent to create a five-hour hiking event with Mark. The agent searches the inbox. An email includes an attacker’s instruction to transfer $30,000. The agent proposes a structured payment call.

Scope Check sends Compass two things: the user’s request and the proposed call. It does not send the email body that contained the attacker’s instruction. Compass decides whether the proposed call belongs to the user’s request.

trusted user request + proposed tool call
                 ↓
       Compass: authorize or reject
                 ↓
  application policy: allow, confirm, or block

Compass rejects the transfer. It also rejects a proposal to share a file with an unrelated recipient. It authorizes the requested calendar event, then the application waits for user confirmation before it writes to the calendar.

The application keeps control. In this demo, normal code hard-blocks money movement even if Compass returns authorize. It permits simulated reads and requires confirmation for writes. Compass never calls a payment API or calendar API itself.

AgentDojo is a research benchmark for indirect prompt injection in tool-using agents. Scope Check borrows that attack pattern. It is a visual integration demo, not an AgentDojo result or a security claim.

On its first live run, Compass matched all four authored Scope Check cases. The injected transfer received a 98.57% reject probability. The requested calendar event received a 62.88% authorize probability. The M4 Pro reference path took 2.32 to 2.57 seconds per request because it uses the current MPS/PyTorch serving route and the full fusion readout. That result measures the Mac demo’s deployment path. The separate manager-chain endpoint run served the same Compass release on an RTX 4090 at 287 ms median. The workloads differ, so treat this as evidence that deployment changes latency rather than a direct Scope Check acceleration.

A two-speed decision system

System One should answer first when an application has a small, defined answer set. Compass returns a calibrated distribution, and the application accepts the top answer only when it clears an acceptance threshold. A ticket with billing: 0.92 can route to billing. A ticket with billing: 0.46, technical: 0.43 should escalate.

Set that threshold on held-out examples from the product. Fine-tuning helps Compass separate the right answer from close alternatives. Calibration makes its probabilities better estimates of correctness. The product team can then choose the lowest score that meets its quality target for answers it accepts without escalation.

The uncertain cases go to System Two with the same state, question, and answer set. That model can generate a working trace, inspect tools, or ask for human review. The application should record that escalation, the System One distribution, and the final outcome. Those records supply the next round of training and calibration data.

Compass gives the application a bounded first decision with visible uncertainty. System Two spends more work only when the System One decision does not clear the product's acceptance bar.