Sweep the thresholds

Eight messages across five tickets are decided and journaled, and each decision is labelled with whether the on-call engineer really paged. A ticket is the hysteresis key, so its follow-ups see the outcome it last got. Then sweep replays the journal through every enter from 0.5 to 0.9 against every exit from 0.3 to 0.9, and prints one row per pair: how many decisions would have been page, how many times a ticket would have flipped between page and wait, and precision, recall and f1 against the labels. The sweep spends no tokens; only the eight decisions do.

// Sweep: which threshold to pick, from the journal. Decisions are journaled and labelled with
// what actually happened; then every candidate enter and exit is replayed over the journal, with
// the flap cost of each pair and precision and recall against the labels. The sweep spends no
// tokens; only the decisions that fill the journal do.

import { mkdtemp } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { fileJournal, fileLabels, huncho, jev, noul, readJournal, sweep } from "huncho";

const dir = await mkdtemp(join(tmpdir(), "huncho-"));
const journal = fileJournal(join(dir, "decisions.jsonl"));
const labels = fileLabels(join(dir, "labels.jsonl"));

const route = huncho("support.route", { model: jev(), journal })
  .ask({ urgent: noul("Does this need a human within the hour?") })
  .when((a) => a.urgent.p, { enter: 0.7, exit: 0.5 }, "page")
  .else("wait");

// Messages arrive over time; a ticket is the hysteresis key, so its follow-ups see the outcome it
// last got. `paged` is what the on-call engineer actually did, known only after the fact.
const messages = [
  { key: "T-4101", body: "Production database is refusing connections. Nothing works.", paged: true },
  { key: "T-4102", body: "The export button is greyed out for one user on Safari.", paged: false },
  { key: "T-4101", body: "Update: connections are back for most customers, a few still failing.", paged: true },
  { key: "T-4103", body: "Login is slow, maybe ten seconds, for everyone in the EU region.", paged: true },
  { key: "T-4101", body: "Update: all connections restored, monitoring for an hour.", paged: true },
  { key: "T-4104", body: "Can we get dark mode? Not urgent, just asking.", paged: false },
  { key: "T-4103", body: "Update: login is back to normal speed in the EU.", paged: false },
  { key: "T-4105", body: "A customer says they were charged twice and wants a refund today.", paged: false },
];

for (const message of messages) {
  const decision = await route.decide(message.body, { key: message.key });
  console.log(`${message.key} urgent=${decision.answers.urgent.p.toFixed(2)} -> ${decision.outcome} (${decision.via})`);
  await labels.write({ id: decision.id, t: new Date().toISOString(), truth: message.paged });
}

// Every enter from 0.5 to 0.9 against every exit from 0.3 to 0.9, the pairs with exit above enter
// left out, plus today's { enter: 0.7, exit: 0.5 } marked current. One replay per pair, no model call.
const result = await sweep(await readJournal(join(dir, "decisions.jsonl")), route, {
  outcome: "page",
  enter: { from: 0.5, to: 0.9, step: 0.1 },
  exit: { from: 0.3, to: 0.9, step: 0.1 },
  labels,
});

console.log("\n  enter exit  chosen flaps precision recall f1");
for (const row of result.rows) {
  const mark = row.current ? "*" : " ";
  const score = (value: number | undefined) => (value === undefined ? "  -  " : value.toFixed(2).padStart(5));
  console.log(
    `${mark} ${row.enter.toFixed(1)}   ${row.exit.toFixed(1)}   ${String(row.chosen).padStart(2)}/${row.n}   ${String(row.flaps).padStart(2)}    ${score(row.precision)}    ${score(row.recall)} ${score(row.f1)}`,
  );
}
if (result.best !== undefined) {
  console.log(`\nbest by f1, ties to fewer flaps: enter=${result.best.enter} exit=${result.best.exit} (today: enter=0.7 exit=0.5)`);
}

Run it

Node 22.18 or later runs the file as it is, after npm i huncho:

TYPESAFE_API_KEY=... node sweep.ts

One line per message prints the urgency, the outcome, and how it was reached: enter when the clause fired on its own, hold when the ticket kept page because the value was still above exit, else when it fell through to wait. Then the table, one row per pair with exit at most enter, the row marked * being today’s { enter: 0.7, exit: 0.5 }, and last the pair with the best f1. A real run, table trimmed to the rows that change:

T-4101 urgent=0.93 -> page (enter)
T-4102 urgent=0.32 -> wait (else)
T-4101 urgent=0.58 -> page (hold)
T-4103 urgent=0.55 -> wait (else)
T-4101 urgent=0.34 -> wait (else)
T-4104 urgent=0.06 -> wait (else)
T-4103 urgent=0.16 -> wait (else)
T-4105 urgent=0.66 -> wait (else)

  enter exit  chosen flaps precision recall f1
  0.5   0.3    5/8    1     0.80     1.00  0.89
  0.5   0.5    4/8    2     0.75     0.75  0.75
  0.6   0.3    4/8    0     0.75     0.75  0.75
  0.7   0.3    3/8    0     1.00     0.75  0.86
* 0.7   0.5    2/8    1     1.00     0.50  0.67
  0.7   0.7    1/8    1     1.00     0.25  0.40
  0.9   0.3    3/8    0     1.00     0.75  0.86

best by f1, ties to fewer flaps: enter=0.5 exit=0.3 (today: enter=0.7 exit=0.5)

Today’s pair never pages wrongly but misses half the real pages. The best row trades one false page for catching all four, and the 0.7 / 0.3 row is the compromise with no flaps at all. The model’s urgency values move a little between runs, and a value near a threshold can land on the other side of it, so your run may differ in outcomes, counts and metrics as well as digits, and best may be a different pair. The run is illustrative; the way to read the table is what carries over.

Read the table in two directions. Down a column of equal enter, lowering exit holds page longer on a ticket once it is there: flaps falls and chosen rises, and recall tends to rise with it while precision tends to fall. Along a row of equal exit, raising enter asks for more certainty before paging at all. best is the pair that would have scored highest on these eight decisions; with eight it is a demonstration, and with a few hundred it is the answer to “what should enter be?” read straight from the journal.

Sweep is what each column means, how a label’s truth is read as “should have been this outcome”, and how best is chosen. Calibration is where the labels come from, and journal and replay is the replay each row runs.

Sweep the thresholds

Run it

Read more