Label with a model. Keep what it said separate.

None of it fails loudly. You get a plausible label and no error.

v1 looked fine. Its own database said otherwise.

1. what v1 recordedwhat v2 does
2. 01Failures became labels.A reply with no label, or a failed request, was saved as the task’s first label; a reply naming two labels counted as whichever came first in the list.A failed or unparseable call is an error row that keeps the raw reply. The parser accepts exactly one allowed label, or entities it can place on the text.
3. 02A confidence nobody measured.Drafts the platform wrote carried 0.9 whatever the model said: 2,274 of them.Drafts store no confidence: none of the providers reports one, and v2 doesn’t invent it.
4. 03Duplicate drafts.479 surplus drafts, and on 99 items they disagreed, so “the model’s label” depended on which row a query returned.Partial unique indexes allow one draft per item, one imported label per item, one answer per person per item.
5. 04Offsets that counted the badges.NER offsets came from a DOM range that included the label badges drawn inside spans: 3 of 1,559 stored spans pointed at the wrong characters.Offsets come from data attributes on each character, in code points, and the server re-checks them and derives the span text itself.
6. 05Saving the assignment dialog moved people’s work.It showed every assignment as items 1…N and wrote that back, so one save could move an annotator from 201–400 to 1–200 and take those items from whoever held them.Ranges are stored and shown as ranges. They limit who is served what; exclusivity comes from claims.
7. 06Exports weren’t filtered.Its README promised human-only and model-only exports; the endpoint wrote every annotation of every source into one file.Every export names its label source: final, human, llm, import or all.
8. 07No claims.Where items were not assigned, everyone drew from one pool, and two people could be served the same item.Items are claimed under a lease with FOR UPDATE SKIP LOCKED.

A correction to v1’s README. It said 3,063 samples went through the review queue a sample at a time. v1 stored every non-draft label as a human annotation, including gold labels written by import scripts. Counting only those created at a human pace, fewer than 100 in the same minute for the same task, leaves 91. The other 94,121 arrived in bulk and are migrated as imported labels, not human ones. The rule and the full reconciliation.

Four sources, and nothing merges them.

The blind audit. A fixed share of claims hides the draft, chosen by a hash of project, item and user so it is stable. Agreement with the draft is then reported separately for shown and hidden claims. If annotators agree with the model far more often when they can see it, the review loop is measuring deference, not correctness.

Two people never get the same item by accident.

WITH candidate AS (
  SELECT i.id FROM items i
  WHERE <eligible for this user>
  ORDER BY i.seq LIMIT 1
  FOR UPDATE OF i SKIP LOCKED
)
INSERT INTO annotations (…, status, lease_expires_at)
SELECT …, 'claimed', now() + lease FROM candidate
ON CONFLICT (item_id, user_id)
  WHERE source = 'human' DO NOTHING
RETURNING id, item_id;

The rest of the workflow.

A seeded demo instance, in both themes and both languages.

Template-generated sentences, simulated annotators, each with a fixed accuracy and copying the draft some of the time when it is shown, and drafts from a real local model, qwen3:4b. The numbers in these pictures describe that simulation and nothing else; templated text is easy, which is why a student can score 100%. npm run screenshots retakes them from a running demo instance.

What was tested here, and what wasn’t.

Node, and nothing else.

$ npm install
$ npm run dev           # API :4000, web :5173

# a demo instance in an empty data directory
$ DATA_DIR=./data/demo npm run seed:demo

# one process serving both
$ npm run build && npm start   # :4000

$ npm test              # 107 tests