3 min read
Night school, eval, coaching. How a digital employee went from 90 to 97 in an hour
Using a data analyst employee as the example, we walk the loop as it exists in the product. Generate a question bank by profession, score a baseline with a judge model, coach a candidate on a snapshot, and let a human decide before it enters version history.
- night school
- eval
- coaching
- walkthrough
This post accompanies the 1 minute 48 second recording on the homepage. The subject is Shen Tunan, a data analyst employee responsible for leads, conversion funnels and weekly business reports.
Step one: night school sets the questions
Open the Eval tab and click Generate. The examiner model first researches the profession named in the soul profile, then writes a syllabus: skill dimensions by question type by difficulty, with a question count in each cell.
Questions are anchored at three levels:
- Executable verification: calculations and reconciliations, deterministic to grade.
- Retrieval anchoring: the reference answer must cite a real retrieved source. Questions without one are discarded, not downgraded.
- Review checklist: open deliverables are ticked against a rubric, with human spot checks.
Generated questions land directly in the bank, split into a practice pool and a test pool. Every question can be edited, moved or disabled. That is the entry point for human review.
Step two: eval sets the baseline
Click Run eval. The system uses the frozen soul of the current version as the system prompt, answers each question, and a judge model scores each answer from 0 to 100 against the reference with a one-line comment. The answering model never scores itself.
Shen Tunan's baseline on ten questions was 90. The comments show where points were lost: some answers were expansive but missed the core phrasing of the reference.
Step three: coaching
Switch to the Coach tab and click Start. Coaching runs on a snapshot. It first re-scores the current soul on the eval set, then hands the trainer model each question, reference answer, baseline excerpt and comment, asks for a rationale and a complete candidate soul, and scores the candidate with the same judge.
About three minutes later the result card appears: 90 to 97. The rationale reads, in short, that the candidate keeps the rigorous style while adding scorable hard constraints: state the standard definition first, the five-stage pipeline, threshold handling, visualization preferences, attribution monitoring. Baseline soul on the left, candidate on the right, comparable line by line.
There is one rule: the candidate must score strictly higher than the baseline. Otherwise it is rejected automatically and the live employee is untouched.
Step four: you decide
Click Apply. The candidate is written through the single soul write path. The Versions tab gains a v2 marked as coaching, with the score change and run id. The soul revision history keeps the full text of v1, and any version can be restored.
The next day Shen Tunan works with v2, and the loop starts again.
A few honest boundaries
- Night school and coaching are started by the user today. They do not run automatically at night. The nightly job only consolidates memory, reviews notes and updates the user profile.
- Eval currently has the model answer directly. Employees do not yet sit exams as a full agent loop inside the sandbox.
- Coaching changes the soul only. Not skills, not memory, not weights.