5 min read
Night school: why a digital employee needs its own exam system
A digital employee cannot grow on user feedback alone. Night school generates a syllabus and a tiered, anchored question bank per profession, separates the examiner from the student, splits practice and hidden pools, and turns eval scores into the external anchor for coaching. This post explains the design choices behind it.
- night school
- evaluation
- design
Once a digital employee is on the job, the hardest question is not "what did it do today" but "will it do better tomorrow". To answer that, you first need a ruler. Night school is where the ruler comes from: it generates a syllabus and a question bank from the employee's profession, so evaluation has a basis and coaching has a target. This post walks through the design choices behind it.
Why not wait for user feedback
The most obvious growth signal is user feedback: the user is unhappy, the employee adjusts. We did not make that the main engine, for three reasons.
First, many users are not fluent with AI. They ask vague questions and give sparse feedback. That rarely converts into an actionable improvement goal.
Second, feedback only covers tasks the user happened to bring. A capability nobody asked about is never measured.
Third, by the time feedback arrives, the mistake has already happened. For business users that cost is real.
So the engine of growth has to be the product's own exam system. Run the school proactively; do not wait for the user to speak up. User feedback can feed the question bank later, but it is not the starting point.
From profession to syllabus
Every digital employee's soul profile carries an occupation and a description of duties. Night school takes that as input, searches the web for what the profession actually does day to day, and distills the findings into a syllabus: capability dimensions by question type by difficulty, with a question count in each cell. Question types include Q&A, calculation, document analysis, and charts and reports.
The syllabus is versioned. Each generation produces a new version and older versions are kept. When the occupation changes, the syllabus can be regenerated.
Question generation follows. The question-setter model fills each cell of the syllabus. Every item carries a prompt, a type, a reference answer, scoring points, and a difficulty. Generated items go straight into the question bank alongside manually entered ones. They can be edited, re-answered, or disabled. Humans do not enter questions one by one; they spot-check.
Three anchoring levels
The biggest risk of automatic question generation is an unreliable reference answer. If the answer is something the model wrote from impression, scoring becomes one model grading another model's impressions with no external basis.
We rank answer reliability into three levels by question type, and every item is labeled with its level.
| Level | Question types | Anchor | Reliability |
|---|---|---|---|
| L1 executable verification | Reconciliation, data calculation | The answer is deterministically checkable | Highest |
| L2 retrieval anchored | Factual and procedural Q&A | The reference answer must cite a real retrieved source | Medium |
| L3 rubric checklist | Open deliverables such as analysis documents and charts | No single correct answer; scored by a checklist ticked item by item | Weakest, backed by human spot checks |
Two rules follow.
An L2 item with no valid source is discarded. It is never downgraded to L3. If web search is unavailable, no L2 items are generated at all.
At L3 the judge is not allowed to give a gut-feeling score. The checklist spells out whether the structure is complete, whether the data is genuinely cited, whether the conclusion is supported, and the judge checks each line. This is the weakest level, so human spot checks start here.
Practice pool and hidden pool
The question bank is split into two pools. The practice pool is for day-to-day improvement. The hidden pool is kept away from the student and used only for regression.
The split exists to prevent overfitting. If improvement is done against the exam questions, scores rise while capability may not. Holding some questions back and validating on them after improvement is what separates "learned it" from "memorized it". Items are assigned to pools by a fixed ratio, stratified so both pools share the same distribution of types and difficulty.
Evaluation runs and the coaching gate count both pools. Hiding is aimed at the improvement process, not at the exam.
Examiner, judge, and student are separate
The question-setter runs on a stronger model tier, separate from the chat model bound to the employee. The judge that scores answers is likewise separate from the answering model. The model that answers never scores itself.
This is a hard rule, grounded in a failure mode that recurs across self-improvement research: when generator and judge share a model, they form a self-confirmation loop where scores climb while capability stalls or regresses. Separating examiner, judge, and student gives the system a reference point that does not drift along with the student.
Scoring itself is constrained. The judge compares each answer to a human-reviewable reference answer, gives a score from 0 to 100, and adds a one-line comment explaining what was missed. Every score traces back to a specific item.
How eval scores anchor coaching
With a question bank and scores in place, coaching has something to stand on.
Coaching is triggered by the user. It first answers the questions with the current soul and scores them to get a baseline. It then generates a candidate soul on a snapshot copy and scores it again with the same questions and the same judge. Only a candidate that scores strictly higher than the baseline is shown to the user for approval. Until the user clicks apply, nothing about the employee changes.
Every soul revision is a full snapshot and can be restored at any time. Improvement is not an irreversible operation; it is one entry in a version history.
That is what bounded self-improvement means here: the goal is defined by an external exam, the result is validated by the same exam, every step can be rolled back, and a human makes the final call.
Not yet built
The design describes a few more things that do not exist today. They are listed here to avoid confusion.
- Running exams as full work sessions in the sandbox, so evaluation covers memory, skills, and tool use. Today the model answers directly.
- An automatic remedial loop: turning judge comments into actions such as installing skills or writing memory, re-examining, and converging on a threshold. Today coaching changes only the soul, not skills or memory.
- Nightly automatic syllabus refresh and automatic exams. Today both night school and coaching are started by the user.
These are later stages on the same road. The basic disciplines come first: syllabus, anchoring, pool split, and separation of roles. Automation comes after.