jobhunt-agent v0.4.0: scores I can measure, from an agent that never reads the ad

jobhunt-agent runs every morning on my laptop, searches job boards, scores new ads against my CV and drafts applications for the best ones. The setup post covers how to install it.

v0.3.0 put limits around the unattended run. It ended with a promise: record a scoring baseline, so I can tell whether a change to the prompts makes the scores better or just different. v0.4.0, "measurable judgment", does that, and uses it to check two bigger changes: the agent that holds my CV no longer reads ad text at all, and the fit score comes from a written rubric instead of a number the model picks.

An eval before any change

v0.3.0 shipped 33 real public ads from the Arbeitsagentur job board, labelled for the repo's sample workspace: Lena, a media and communication student in Mannheim looking for marketing and social media roles. 9 ads are in her field near home, 12 are in her field but too far away, and 12 are out of scope (sales, engineering, senior roles, an apprenticeship). Each ad has the answers the routine should give: in scope or not, role type, hard mismatch, an acceptable score range and the German level the ad asks for.

tools/eval_run.py run judges each ad the way the daily run judges a new job, with one headless claude -p call per ad, no tools and an empty folder. The prompt is built from the checkout's own ROUTINE.md, RULES.md and rubric, so a run measures the engine as it is. It writes a baseline per version, and eval_run.py compare fails when any metric drops by more than 2 ads. Model answers vary between runs, so one or two ads is noise.

I recorded the v0.3.0 baseline first, before touching the scoring. Then the v0.4.0 one, after:

Both were run with Claude Opus 5.5. The German level is the real gain: the extraction schema now spells out that "sehr gute Deutschkenntnisse" means C1+ and that an ad stating no level is unknown, and the model stopped guessing. The score drop is two ads I looked at one by one. Ad 02 scores 89 against a range of 70 to 88, one point over. Ad 19 is an online editing job whose location the new rubric reads as a hard mismatch, which pulls its score to 16. Neither changes whether a draft gets written: the tailoring decision agrees on all 30 clear-cut ads, before and after.

The v0.4.0 run cost $3.56 for 33 ads. It's for engine development only, and the release checklist now runs it before every release.

The agent with the tools never reads the ad

In v0.3.0 the run that holds my CV, my Gmail and the shell also read every ad. Permission rules and a guard hook limited what it could do, but a stranger's text still sat in the same context as everything it could do it with.

In v0.4.0 a separate ad-reader subagent reads every ad, Indeed job, Gmail job alert and careers page. Its definition removes the write, shell, agent and send tools, and it doesn't load CLAUDE.md, so it knows nothing about me:

name: ad-reader
disallowedTools: Write, Edit, MultiEdit, NotebookEdit, Bash, PowerShell, Agent, Task, Skill, TodoWrite
omitClaudeMd: true

It returns only JSON in a fixed schema: title, company, location, role type, a closed flag with the words that show it, German level, hours, start, up to 12 requirements with their importance, the tasks, and injection_text for any instructions it found in the ad, described in a few neutral words rather than quoted. extract.py save checks every reply and refuses links, email addresses and AI instructions in any field except url and injection_text. The routine judges, scores and tailors from that extract. Each job folder gets ad.json and ad.txt, and the routine is told never to open the text.

The worst an ad can now do is lie about its own facts. It can't reach the context that holds the CV and the tools. The cost is one extra model call per ad, about $0.05.

A rubric instead of a number

Until now the model looked at an ad and my CV and picked a fit score from 0 to 100. Scoring the same ad twice could give different numbers, and nothing said what a 74 meant.

engine/rubric.md now has written anchors from 0 to 5 for five criteria: skills, seniority, language, location and hours (with start date, duration and enrolment). The model picks the anchor that fits, and tools/score.py does the arithmetic. Skills sets the base, each other criterion below 5 takes points off, and a 0 for seniority, language, location or hours is a hard mismatch that halves the total:

$ python engine/tools/score.py skills=4 seniority=5 language=3 location=5 hours=4
score 81  hard mismatch: none
  skills 4/5: base 88; seniority 5/5; language 3/5: -5; location 5/5; hours 4/5: -2

The same job in a city Lena can't reach, location=0, scores 28. A hard mismatch always lands below 60, under every threshold the routine acts on. The levels were set against the labelled ads, and config.yaml can change the weights, which criteria are blockers and the bonus points for skills I want to use. The same ad scored twice stayed within 3 points.

Each job.md now lists the five sub-scores with their reasons, so when a score looks wrong I can see which judgment to disagree with.

Jobs that came back, and jobs that should

• Ads the reader couldn't read. A bot check or an empty page used to be recorded in the ledger as screened, so the job never showed up again. Now it's listed as "couldn't be read" and tried again on the next run. A page that's gone (404, 410) counts as closed.
• Ledger expiry. A job screened, skipped or closed more than 90 days ago counts as new again if it's reposted or still online. Jobs I tailored, applied to, or got an interview or rejection for never expire. ledger_expiry_days changes the days, and 0 turns it off.

One fix came from my own workspace. v0.3.0's permission template denied the Agent tool, which kept the daily run from calling the ad reader, and setup_check.py didn't notice. The migration now saves .claude/settings.json as a dated backup, removes that one rule and prints removed deny Agent.

Upgrading

Open the workspace folder in the Code tab and say:

update the engine

The step that's new in v0.4.0 is the migration: config.yaml stays at version 2, .claude/agents/ad-reader.md is copied into the workspace, and a deny rule for Agent is removed. setup_check.py warns until both are done. v0.4.0 doesn't change the document builder, so the comparison should say SAME DOCUMENTS.

Then one Run now, and three checks:

1. The brief's Sources section shows engine 0.4.0 (…) and guard active.
2. data/ads/ has a .json file for each new job, and each tailored job's job.md lists its five sub-scores.
3. Jobs the brief lists as "couldn't be read" aren't in data/seen_jobs.csv.

Scores can move by several points against older briefs. That's the rubric, not a bug. python engine/tools/upgrade.py rollback puts the old engine back. The upgrade guide has every step, and the changelog lists every behaviour change.

What's next

v0.5.0, "grounded & modular": every letter is checked against its sources before it's built, and each job is tailored in a fresh context. Now that there's a baseline, I'll know whether that makes the drafts better or just different.