Project 02. The Readiness Scorecard

Illustrative composite. Ledgerline is a made-up five-person company that sells bookkeeping software to small shops. On Friday its lead wrote in the team chat that the repo was "mostly there" and that agents would start pulling tickets from the app/ queue on Monday. Nobody had run a single readiness test. When someone finally counted, seven of the eight checks passed, and seven of eight feels like a pass. The one that failed decides whether an agent can build the code at all. Nobody had noticed because every person on the team builds it on a laptop that can reach the internal package index. The sandbox the agent works in cannot.

You have a readiness list from Lesson 9, a blueprint from Lesson 8 and a release path from Lesson 10. This project makes them meet on one repo: score it, mark its line as deterministic and agent nodes, attach the way its changes reach customers, and write down the tier it earns. The deliverable is factory/readiness-scorecard.md - a repo scored against the readiness list, with the blueprint nodes and release path attached, and the tier it earns. It serves the Run stage of the course loop (Slice, Gate, Run, Wrap, Prove): it decides how much of the line may run without a person watching, and what you fix first to raise that.

Brief

Use the fictional repo below, ledgerline-api. Every input you need is on this page. If you want to score your own repo afterwards, the steps are the same, but do the fixture first: the file you write here is checked against it. Do not score from memory or from the chat thread. Every result in your table needs a line of evidence taken from the fixture, in the same way your real table would carry a command output.

Illustrative composite. Every name, host, ticket and figure in this Brief is made up. Ledgerline does not exist.

The repo

ledgerline-api   (Python service, five engineers, one platform lead)Top-level paths  app/          invoicing and refund API  docs/         developer docs site  migrations/   database migrations  tests/        unit and contract tests  scripts/      release and deploy helpers  .github/      workflows and templatesTeams that exist in the org  @ledgerline/platform   4 members  @ledgerline/data-eng   2 members  @ledgerline/support    3 membersCODEOWNERS  *             @ledgerline/platform  /migrations/  @ledgerline/data-eng  /docs/        @ledgerline/platformREADME.md headings  ## What it does  ## Run locally       (step 1: docker compose up -d db; step 2: pip install -r requirements.txt)  ## Run the tests     (make test).github/pull_request_template.md  "## What changed", "## How it was tested", and four checkbox lines  (tests added, docs updated, migration reviewed, rollback noted).github/workflows/ci.yml  on: pull_request and push to main; steps: lint, make test

The evidence you were handed

$ gh run list --branch main --limit 10      (event, conclusion, attempt)  push  success  1  push  success  1  push  success  1  push  success  1  push  success  1  push  success  1  push  success  1  push  success  1  push  success  1  push  success  1Fresh container, README steps 1 and 2 only, then make test:  212 passed, 0 skipped, exit 0Sandbox build: fresh sandbox, network limited to the hosts indocs/allowed-hosts.md (pypi.org, files.pythonhosted.org). README "Run locally"followed literally. requirements.txt contains:  --extra-index-url https://pkgs.ledgerline.example/simple  ledgerline-money==2.3Result:  ERROR: connection to pkgs.ledgerline.example blocked  ERROR: no matching distribution found for ledgerline-money==2.3  exit 1README does not name pkgs.ledgerline.example. docs/allowed-hosts.md does not list it.Five open tickets sampled from the app/ queue  LL-412  "Rounding error on 3-line invoices: expected total 10.00, got 9.99; see app/invoices/rounding.py"  LL-415  "POST /refunds returns 500 on a zero amount; expected 422 with a message; handler in app/refunds/api.py"  LL-419  "Invoice PDF drops the VAT line; expected a VAT line after the subtotal; template app/invoices/pdf.py"  LL-421  "Refund bug, see the thread in #billing"  LL-423  "Credit note date shows UTC; expected shop-local date; app/credit_notes/render.py"

The rules you apply (from Lesson 9)

The eight checks and their one-minute tests:

1 Named owner       Bronze  CODEOWNERS has a rule for every top-level path, each naming a person or team that exists2 README            Bronze  one section each for what it does, how to run it, how to test it3 PR template       Bronze  .github/pull_request_template.md exists and holds a checklist4 CI workflow       Bronze  a workflow triggers on pull_request and runs build and tests5 CI green          Bronze  the last ten runs on the default branch all passed on their first attempt6 Runnable tests    Silver  from a fresh clone, make test exits 0 with no step beyond the README7 Build steps       Silver  in a fresh sandbox limited to docs/allowed-hosts.md, README build steps followed literally exit 08 Tickets           Gold    sample five open tickets; each states expected behaviour, a file or entry point, and more than a link; at least four pass

The tiers. A repo holds the highest tier whose checks all pass, and Gold needs Silver:

The team's autonomy map from Lesson 3:

The tier cap per lane: at Bronze, docs and tests are capped at light (draft PRs, a human assigns, approves and merges) and app and migrations at human-led. At Silver, app may rise to light; migrations stay human-led. The stricter of the map level and the tier cap wins. The ladder runs dark, then light, then human-led, from loosest to strictest.

The line the team drew (not yet a blueprint)

1. A planner model reads the backlog each morning and picks up to five tickets.2. The coding agent implements each ticket in a sandbox.3. The agent runs the tests and writes "tests pass" in the PR description.4. A linter runs when the agent pushes.5. CI runs. If it is red, the agent keeps fixing until it is green or six hours pass.6. A reviewer model comments on the PR.7. A script merges the PR when the reviewer comment does not contain the word "blocking".8. The merge deploys to staging, then a nightly job ships main to production.

The release path today

Merge target   PRs merge to main. main ships to production by itself at 02:00 UTC every night.Staging        exists; deployed by hand from any branch with `make deploy-staging`. Nobody is assigned to look at it.Canary         the feature-flag service can send a configurable share of traffic to a new build.               It has never been used for a release. No checks are written.Rollback       `make rollback` redeploys the previous tag. Never tried. No named owner.Human release  none. The 02:00 job ships whatever is on main.

The facts for the tier-drop rule

• Approver: Mara Okafor, platform lead. Backup: Tomasz Weiss, data-eng lead.
• The default-branch workflow has a known flaky integration test that has failed once and passed on rerun on a handful of runs over the past month. A single red run does not mean the repo slipped.
• A nightly job can run checks 1 to 7. A weekly job can re-sample check 8.
• Agent identities have write access to the repo through pull requests only, never to the ruleset or to factory/.

The scorecard template

Steps

Step 2 is worked for you on one check, so you can see the grain of evidence expected. Step 3 holds one judgment call with the answer beneath it. Steps 4 to 7 are open: you have the template, the rules and the fixture.

1. Create the skeleton. Copy the scorecard template into factory/readiness-scorecard.md and fill the Scope section. Produces factory/readiness-scorecard.md with seven headings, spelled exactly as in the template. The scorer is a named person. If the Scope section says an agent scored the repo, start the file again.

2. Score the eight checks. For each check, write PASS or FAIL and one line of evidence taken from the fixture, not from your impression. A worked row:

   markdown

   previewsource

   copy

   | 1 | Named owner | CODEOWNERS has a rule for every top-level path, each naming a person or team that exists | Bronze | PASS | * rule covers every path; targets @ledgerline/platform and @ledgerline/data-eng both exist in the org |

   Do the other seven rows yourself. Produces the Checks table in factory/readiness-scorecard.md, eight rows. Evidence names what you looked at (a path, an output line, a ticket number). "Looks fine" is not evidence.

3. Set the tier and cap the lanes. Write one sentence naming the held tier and the check that stops the next one, then fill the lane table: autonomy map level, tier cap, effective level (the stricter). Fill the Held tier: and Blocked: lines in Scope to match. Produces the Tier earned section, four lane rows.

4. Turn the team's line into a blueprint. Rewrite the eight steps as nodes, one row each, in the table from Lesson 8 (use more rows than eight where a step splits). Mark each node deterministic, agent or human. Every agent node names a node other than itself as the one that decides it is done, a retry cap and an exit. Add a node that reads the held tier from this file before dispatch, and a node that fails the task when the branch has no commits. A filled row to start from:

   markdown

   previewsource

   copy

   | 1 | Dispatch | deterministic | Reads issues with the label agent-ok; only a platform-team member can apply it | Code: a run record exists | none | A duplicate or unlabelled trigger is dropped |

   Where the draft breaks a Lesson 8 rule (a model picking tickets, an agent reporting its own result, a loop with a clock instead of a count, a script merging on a word match), change the node and say in the exit column what the new exit is. Produces the Blueprint nodes section, at least nine rows.

5. Attach the release path. Fill the table: merge target, staging validation, canary, rollback, human release step, one row each at minimum. Name an owner for each, and for the rollback row name the person who cut the release, not the agent. Put in the tier that has to be held before that phase may run unattended. For the canary row, state which checks you would write and how you will count each check's false positives and when a noisy check is demoted; the number and window are your design, and you say so in the row. Replace the 02:00 auto-ship with a human step. Produces the Release path section, five or more rows.

6. Write the tier-drop rule. Number the rule so a stranger can follow it: which job re-runs which checks and how often; how many consecutive failures, at least an hour apart, count as a real failure (the flaky test is why one red run must not count); what the job may do alone (open an issue and add the label readiness-freeze, nothing else); which lanes freeze and what happens to open PRs; who approves, who covers when they are away, and why silence never lifts the freeze; how the tier is restored and by whom; and that no agent may edit this file or the job's config. Then write the issue template the job opens, with a title that begins PROPOSED:, and fields for the failed check, the evidence link, first-failed and confirmed-failed times, frozen lanes, in-flight agent PRs with their reviewers, the approver, the fix issue and a Restore when: line. Produces the Tier-drop rule section.

7. Name the first fix and check the file. Write the one job that raises the tier: what the issue changes, which lane it is in (it must be a lane the held tier unlocks), who assigns it, and the deterministic test that accepts it. Then run this from the repo root, and fix whatever it prints:

   bash

   copy

   f=factory/readiness-scorecard.mdfor h in Scope Checks "Tier earned" "Blueprint nodes" "Release path" "Tier-drop rule" "First fix"; do  grep -q "^## $h\$" "$f" || echo "MISSING HEADING: $h"doneawk -F'|' '/^## /{on=($0=="## Checks")} on && /^\| [1-8] \|/ { n++; if ($6 !~ /^ *(PASS|FAIL) *$/ || $7 ~ /^ *$/) print "BAD ROW: " $2 } END { if (n < 8) print "ONLY " n " CHECK ROWS" }' "$f"grep -q 'readiness-freeze' "$f" || echo "NO FREEZE LABEL"grep -q '^Held tier\|Held tier:' "$f" || echo "NO HELD TIER LINE"

   Also add factory/readiness-scorecard.md to factory/protected-paths.md. Produces the First fix section and the protected-path entry. A clean run prints nothing.

8. Hand it to a second reader. Give someone only the finished file and ask: which tier does this repo hold, what blocks the next one, and what may the agent do on Monday? If their answers differ from yours, edit the file, not your explanation.

Acceptance

• factory/readiness-scorecard.md exists and has the seven headings Scope, Checks, Tier earned, Blueprint nodes, Release path, Tier-drop rule and First fix, spelled exactly.
• The Checks table has eight rows, each with PASS or FAIL and a non-empty evidence cell; the step 7 script prints nothing.
• Checks 1 to 6 and 8 are PASS, check 7 is FAIL, and the evidence for check 7 names the blocked host.
• The held tier is Bronze, the Blocked: line names check 7, and no sentence in the file says the repo is "close to" Silver or Gold.
• The lane table has four rows, and no lane's effective level is looser than either the autonomy map or the tier cap.
• The Blueprint nodes table has at least nine rows; dispatch, the tier read, the commit check and CI are deterministic; no agent node decides its own completion.
• Every agent node has a retry cap and an exit, and at least one exit names a human.
• The merge node is a human node at Bronze, and the 02:00 auto-ship does not appear as a step.
• The Release path table has a named owner on every row, the rollback owner is the person who cut the release, and the canary row names how false positives are counted.
• The tier-drop rule requires two failures in a row at least an hour apart, limits the job to opening the issue and adding readiness-freeze, and says silence never lifts the freeze.
• The approver, the backup and the restore step are named, and the file lists itself as a path no agent may change.
• First fix is a docs-lane job, assigned by a human, accepted by a sandbox build that exits 0, and it includes no credentials.
• factory/protected-paths.md lists this file.
• A second reader answers the three questions in step 8 the way you did.

Rubric

Key Takeaways

• Score a repo from evidence in hand; never from the team's confidence.
• Hold the highest tier whose checks all pass, because passing seven of eight proves nothing about the one that blocks.
• Cap each lane at the stricter of the autonomy map and the held tier.
• Rewrite every step where a model decides its own result into a node something else decides.
• Give the release path a human step and a rollback owner who is not the agent.
• Let the tier-drop job propose and freeze; let a named person decide.

What You Have Now

factory/readiness-scorecard.md - a repo scored against the readiness list, with the blueprint nodes and release path attached, and the tier it earns. It sits in factory/ beside factory/readiness-score.md from Lesson 9, whose checks it applies; factory/blueprint.md from Lesson 8, whose node table it fills for one line; and factory/release-path.md from Lesson 10, whose phases it attaches. You can now say what a repo may do unattended on Monday, what stops the next tier, and who decides when it slips. Project 3 reads this file when it reconciles your module files into a single charter, and the held tier you wrote here caps the first lane that charter lets you switch on.