Change it every day without breaking trust
Every change takes the same checked path: tests and evals in CI, a review, then a staged rollout you can undo in seconds.
By the end of tonight you can
- Set up continuous delivery: branch, pull request, CI, review, deploy
- Run tests and evals on every pull request, with branch protection blocking a failing merge
- Put Claude in the pipeline: @claude reviews and Claude Code in GitHub Actions
- Build a dashboard for errors, speed, AI quality and cost, with alerts that name an owner
- Plan a staged rollout behind a feature flag (5%, 25%, 100%) with a rehearsed rollback
OUTPUT · CI pipeline + dashboard + staged rollout plan in product/11-operations.md
Continuous delivery: one path for every change
- BRANCHEvery change starts on its own branch; main always holds what is safe to ship
- PULL REQUESTPropose the change and say what changed, why, and how you checked it
- CITests and evals run automatically; with branch protection, a red check blocks merging
- REVIEWRead the diff with Week 5’s five questions; Claude can review too, a person merges
- DEPLOYMerging deploys to staging automatically; production follows a quick check
Small changes, the same path every time. The path, not heroics, is what makes daily change safe.
Claude in the pipeline
- @CLAUDEComment “@claude review this for access control and missing tests” on a pull request
- GITHUB ACTIONSClaude Code runs as a workflow step, e.g. reviewing every pull request unasked
- EVALS ON EVERY PRYour Week 10 harness runs in CI; below the agreed score, the check fails
- KEYS AND COSTThe API key is a GitHub secret; eval runs count against your spend limit
- A PERSON DECIDESClaude’s comments are advice; a person owns the merge button
Claude reviews every change; the evals check Claude itself. You still decide what merges.
Observability: know what tutors are experiencing
- ERRORSErrors per hour, grouped by type, each with a request ID you can trace
- SPEEDHow long pages and Claude calls take at the slow end (95th percentile), not the average
- AI QUALITYShare of proposals tutors approve as-is, edit or reject; how often the fallback fires
- COSTTokens and spend per day and per active tutor, against your Week 8 estimate
- ALERTSA few alerts on what users feel, each with an owner and a first step
Watch what tutors experience and what it costs. An alert nobody acts on is noise.
A prompt change, caught before merge
“I only made the replies friendlier. It’s one sentence in the system prompt; it can’t break anything.”
The eval check on the pull request falls from 17 to 14 of 20 cases, below the agreed 16. Three proposals now offer dates the parent ruled out. CI blocks the merge; the fix keeps the warmth.
Ask
- What would tutors have seen if this had merged on Friday?
- Is 17 to 14 a real drop or noise? How would you tell?
- Which new eval case would you add because of this?
- Who, if anyone, may override a failing eval check?
Scaling a measured bottleneck vs an imagined one
An imagined bottleneck
- “We’ll need a new architecture when we reach 10,000 tutors”
- Weeks of rewriting before anyone is waiting
- No number shows the problem, so no number shows it fixed
A measured bottleneck
- The dashboard: proposals take 9 seconds at the 95th percentile (illustrative)
- Timing each step: 7 of those seconds are one Claude call with a long prompt
- One lever, such as prompt caching or streaming, then measure again
Measure, fix the slowest step, measure again. TutorDesk has about 20 tutors, not 20,000.
Feature flags and staged rollout
- 5%The new version reaches a few tutors who agreed to try it; watch errors, approvals, cost
- 25%Widen only if the guardrail numbers held for the agreed time, e.g. a week
- 100%Everyone has it; once stable, remove the old path and the flag
- ROLLBACKSwitch the flag off in seconds, no deploy; agree the stop rule before you start
Decide the stop rule before you start. A rollback you have never tried is a hope, not a plan.
CI that runs tests and evals on every pull request
I’m building TutorDesk, and the code is on GitHub. My tests use a fake Claude response; the eval harness in evals/ calls the real API and prints a score like “17 of 20”. I want every pull request checked automatically so that a code or prompt change can’t quietly lower quality. Plan first, then write a GitHub Actions workflow that runs the tests on every pull request, and runs the evals when files in [PROMPT AND AI FOLDERS] change. The eval step should fail below [THRESHOLD] of 20 and list the failing case IDs. Read the API key from a repository secret. Then tell me the cost of one eval run from the token counts, and the branch protection settings I need. If you’re unsure how my harness reports results, ask me rather than guess.
Check the response
- A red eval check really blocks the merge
- The key is a secret, never in the workflow file
- Cost per eval run is stated with its basis
Workshop: CI with evals, a dashboard, a rollout plan
Give your product a path you can use every day: tests and evals in CI, a dashboard you can read in a minute, and a rollout you can undo.
- Add the CI workflow and turn on branch protection so a failing check blocks the merge
- Open a pull request that deliberately fails one eval case; keep the link to the blocked merge
- Ask @claude to review a real pull request; accept or reject each comment with a reason
- Build a one-page dashboard of errors, slow-end speed, AI quality and cost; set two alerts
- Write the rollout plan (stages, guardrails, stop rule) and rehearse the rollback on staging
CI, dashboard and rollout plan in product/11-operations.md; workflow committed
Operations rubric
- CI runs tests and evals on pull requests; branch protection makes a failing check block the merge
- A linked pull request shows a deliberately failed eval check that blocked the merge
- The eval check reports a score with its base (e.g. 17 of 20) and the failing case IDs
- A pull request has a Claude review, with a recorded decision on every comment
- The dashboard shows errors, speed, AI quality and cost; two alerts each name an owner
- The rollout plan names stages, guardrail numbers, a stop rule and a rehearsed rollback
NEXT · Week 12: go to market. Bring your rollout plan and a week of dashboard numbers.
© 2026 MVP Studios Ltd. MVPAI™ and the MVP Founders Programme materials are original works of MVP Studios Ltd. All rights reserved. No copying, redistribution or commercial use without written permission. Course terms