Labels and calibration

Six tickets are decided and journaled to a file. For each one, what the on-call engineer actually did is written to a second file as a label against the decision’s id. Then calibrate reads the labels, joins them to the records by id, and scores the probabilities against what happened. Nothing is joined by hand, and the labels could have been written days later by another process: both files are append-only JSONL.

// Labels close the loop: journal decisions, record what actually happened for each one by
// its id when the truth arrives, and calibrate with no hand-written join. Both files are
// append-only JSONL, so the labels can be written days later by a different process.

import { mkdtemp } from "node:fs/promises";
import { tmpdir } from "node:os";
import { join } from "node:path";
import { calibrate, fileJournal, fileLabels, huncho, jev, noul, readJournal } from "huncho";

const dir = await mkdtemp(join(tmpdir(), "huncho-"));
const journal = fileJournal(join(dir, "decisions.jsonl"));
const labels = fileLabels(join(dir, "labels.jsonl"));

const route = huncho("support.route", { model: jev(), journal })
  .ask({ urgent: noul("Does this need a human within the hour?") })
  .when((a) => a.urgent.p, { enter: 0.7, exit: 0.5 }, "page")
  .else("wait");

// What the model decided, then what the on-call engineer actually did: `paged` is the truth
// a support desk learns after the fact, from the incident log rather than from the model.
const tickets = [
  { id: "T-3101", body: "Production database is refusing connections. Nothing works.", paged: true },
  { id: "T-3102", body: "The export button is greyed out for one user on Safari.", paged: false },
  { id: "T-3103", body: "A customer says they were charged twice and wants a refund today.", paged: false },
  { id: "T-3104", body: "Can we get dark mode? Not urgent, just asking.", paged: false },
  { id: "T-3105", body: "Login is slow, maybe ten seconds, for everyone in the EU region.", paged: true },
  { id: "T-3106", body: "Password reset emails are not arriving for any customer since noon.", paged: true },
];

for (const ticket of tickets) {
  const decision = await route.decide(ticket.body, { key: ticket.id });
  console.log(`${ticket.id} urgent=${decision.answers.urgent.p.toFixed(2)} -> ${decision.outcome}`);

  // When the truth arrives, label the decision by its id. Only `id`, `t` and `truth` are needed.
  await labels.write({ id: decision.id, t: new Date().toISOString(), truth: ticket.paged });
}

// Later, anywhere the two files are readable: the join by id is calibrate's.
const c = await calibrate(await readJournal(join(dir, "decisions.jsonl")), { question: "urgent", outcome: labels });

console.log(`\nscored ${c.n} decisions: brier=${c.brier.toFixed(3)} baseBrier=${c.baseBrier.toFixed(3)} baseRate=${c.baseRate.toFixed(2)}`);
console.log(c.brier < c.baseBrier ? "the probabilities beat the base rate" : "the probabilities do not beat the base rate");
for (const row of c.reliability) {
  const bin = `[${row.lo.toFixed(1)}, ${row.hi.toFixed(1)}${row.hi === 1 ? "]" : ")"}`; // the last bin includes 1
  console.log(`p in ${bin}: n=${row.n} predicted=${row.meanP.toFixed(2)} observed=${row.observed.toFixed(2)}`);
}

Run it

Node 22.18 or later runs the file as it is, after npm i huncho:

TYPESAFE_API_KEY=... node labels.ts

One line per ticket prints the urgency and the outcome, as the replay example does. Then the summary: how many decisions were scored, the Brier score beside the base-rate Brier score, whether the probabilities beat the base rate, and one reliability row per occupied bin with the mean predicted probability against how often a page really happened. A real run:

T-3101 urgent=0.94 -> page
T-3102 urgent=0.30 -> wait
T-3103 urgent=0.68 -> wait
T-3104 urgent=0.06 -> wait
T-3105 urgent=0.56 -> wait
T-3106 urgent=0.82 -> page

scored 6 decisions: brier=0.131 baseBrier=0.250 baseRate=0.50
the probabilities beat the base rate
p in [0.0, 0.1): n=1 predicted=0.06 observed=0.00
p in [0.3, 0.4): n=1 predicted=0.30 observed=0.00
p in [0.5, 0.6): n=1 predicted=0.56 observed=1.00
p in [0.6, 0.7): n=1 predicted=0.68 observed=0.00
p in [0.8, 0.9): n=1 predicted=0.82 observed=1.00
p in [0.9, 1.0]: n=1 predicted=0.94 observed=1.00

T-3105 is the interesting row: the model gave it 0.56, the policy said wait, and the label says a page really happened. That is the kind of miss calibration exists to surface.

With six tickets the numbers are a demonstration, not a verdict. The shape is what matters: labels.write happens wherever the truth becomes known, calibrate happens wherever both files can be read, and the id is the only thing that has to travel between them.

Calibration is the loop, the label shapes for noul, choice and score, and what the numbers mean. Journal and replay is where the id comes from.

Labels and calibration

Run it

Read more