MVPSTUDIOSIDEAS · PRODUCTS · PEOPLE Start a conversationLet's talk
← Week 10: Assemble at scale: build Claude into your product SESSION 02 · THURSDAY

Prove it is good enough, and affordable

“It worked when I tried it” is not evidence. Tonight you score your feature on agreed cases, price it per user and make it reliable.

EVALSGRADERSCOST PER USERSPEEDRELIABILITY
Present slides Full screen, one slide at a time · 12 slides
TONIGHT 01

By the end of tonight you can

  • Run an eval set end to end and state the score with its base, against a threshold set in advance
  • Choose a grader for each check: exact code, Claude with a written rubric, or a person
  • Compare two prompts (or two model tiers) on score, cost and speed from real usage data
  • Calculate cost per user per month and name the levers that would lower it
  • Add timeouts, retries with back-off, rate-limit handling, a fallback and safe logging

OUTPUT · Eval harness + results + cost per user + reliability checklist in product/evals/

THE IDEA 02

“It worked when I tried it” vs “it passes the eval”

It worked when I tried it

  • Five tries, typed by you, on the cases you happened to think of
  • Judged by feel, so a weaker reply can still look fine
  • Nobody can repeat it after you change the prompt or the model tier

It passes the eval

  • 20 agreed cases, including hard and adversarial ones, written before building
  • Each graded the same way every time, with a reason for every failure
  • Rerun after every change, so a drop in quality shows before users see it

Tests check your code with a fake Claude. Evals check Claude’s real behaviour. You need both.

THE SKILL 03

Running evals

  1. DATASETYour week 8 cases: real, hard and adversarial inputs, each with expected behaviour
  2. RUNSend every case through the product’s own code path; save each reply and its usage
  3. GRADEMark each case pass or fail, with a reason, using the grader that suits it
  4. COMPAREScore against the threshold set before building, and against the last run
  5. DECIDEShip, change one thing and rerun, or stop; write the decision down

Change one thing at a time, then rerun the whole set, not just the cases that failed.

THE SKILL 04

Graders: who decides pass or fail

  1. CODE CHECKSExact and cheap: valid JSON, the slot was offered, no slot on an avoided date
  2. CLAUDE + RUBRICFor judgement: a second call grades the reply against a written rubric, with a reason
  3. HUMAN REVIEWA person judges tone and usefulness, and checks a sample of the other grades
  4. CHEAPEST FIRSTCode wherever it can decide, Claude where judgement is needed, people to check both

Trust a grader only after a person has checked a sample of its grades.

WORKED EXAMPLE · TUTORDESK 05

TutorDesk’s eval run: 17 of 20 (illustrative)

Starting point

“17 out of 20 is 85%. Ship it.”

Reframed

17 clears the bar of 16, but one failure is a safety case: a slot on “not the 14th”. Also: a slot for a cancellation, one slot for two children. Fix, rerun all 20.

Ask

  • Which failure would hurt a tutor’s trust most, and why?
  • Which fix belongs in the prompt, the JSON shape or the code?
  • What new cases do these three failures suggest?
  • What does “85%” hide that the failures show?
COST AND SPEED 06

Cost and speed levers

  1. MODEL TIERTry a smaller, faster tier first; move up only if the eval score needs it
  2. SHORTER PROMPTSCut instructions and data the eval shows Claude doesn’t need: fewer input tokens
  3. PROMPT CACHINGReuse a long, unchanging start of the prompt to cut cost and wait on repeat calls
  4. BATCHINGMessage Batches: many requests together at lower cost, when nobody is waiting
  5. STREAMINGShow the reply as it is written: the same tokens, but the wait feels shorter

Every lever is a trade. Rerun the eval after each one: cheaper and worse is not a saving.

RELIABILITY 07

Reliability: plan for the call failing

  1. TIMEOUTSDecide how long a user should wait, then stop waiting and use the fallback
  2. RETRIESOn a temporary error, wait a moment and retry, waiting longer each time; stop after a few
  3. RATE LIMITSYour account can make only so many calls in a period: queue and slow down
  4. FALLBACKSThe product still works without AI: “we couldn’t read that message, please choose a slot”
  5. LOGGINGRecord time, tokens, outcome and error type; leave out messages and names

When Claude can’t help, the tutor can still book a lesson. That is what reliable means here.

MVPAI · THE ASSEMBLE GATE 08

The loop 2 Assemble gate: demo, then two rails

  1. DEMOReal users try the AI slice on staging; count who reached first value, with a base
  2. EVAL SCOREScore against the threshold set in week 8, e.g. 17 of 20 vs 16 (illustrative)
  3. TESTS AND EVIDENCETests pass with a fake Claude response; what users did in the demo, counted
  4. HUMAN SIGN-OFFA person signs off; a Claude review is one input, never the verdict
  5. MUST-PASSAI safety cases incl. injection, secrets, access control, personal data: all pass

TutorDesk: 17 of 20 with every safety case passing. No score outweighs a failed must-pass check.

CLAUDE STUDIO 09

Build an eval harness and compare two prompts

Prompt for Claude
I’m building TutorDesk’s slot-proposal feature. I need to know whether a shorter prompt is as good and cheaper, because I’ll pick one before week 11. The 20 cases and their expected behaviour are in product/evals/[CASES FILE]; prompts A and B are [PROMPT FILES].
Plan first, then build a script that runs every case through our real code path on staging, grades each with code checks (valid JSON, slot offered, no avoided date) and writes product/evals/results.md: case, pass or fail, reason, input and output tokens from the usage data, seconds. Work out cost with these prices: [PRICES FROM PRICING PAGE].
Finish with each prompt’s score out of 20, total tokens, cost and time. Report the real runs, errors included; if a case needs a person to judge, say so.

Check the response

  • Every case ran; token counts come from the usage data
  • Each failure has a reason; judgement cases go to a person
  • Cost uses the prices you entered, with the date checked
WORKSHOP · YOUR PRODUCT 10

Workshop: eval harness, scored run, one improvement

Build your eval harness with Claude Code, score your feature (or internal tool), improve one thing, price it per user and make it reliable.

  1. Brief Claude Code with the Studio prompt; read and approve its plan, then run all your cases
  2. Check two passes and two failures by hand; fix any grade you disagree with
  3. Change one thing (prompt, data sent or model tier), rerun every case and compare
  4. Work out cost per user per month from real token counts and today’s dated prices
  5. Add a timeout, retries, rate-limit handling and safe logging; show the fallback firing
Deliverable

Harness, results, cost per user and reliability checklist in product/evals/

GATE 11

Evals rubric and the Assemble gate

  • One command runs every case in product/evals/ and writes a results table
  • The score shows its base and the threshold set before building (e.g. 17 of 20 vs 16)
  • Every failure has a written reason, and a person checked at least four grades
  • Two prompts or tiers are compared on score, tokens, cost and time from real usage data
  • Cost per user per month shows its arithmetic, with prices dated from the pricing page
  • Timeout, retries, rate-limit handling, fallback and personal-data-free logging each shown

NEXT · Week 11: make it safe. Bring your harness, your results and messages you suspect could fool Claude.

© 2026 MVP Studios Ltd. MVPAI™ and the MVP Founders Programme materials are original works of MVP Studios Ltd. All rights reserved. No copying, redistribution or commercial use without written permission. Course terms