← Week 07: Map at scale: why people stay, leave and pay
SESSION 02 · THURSDAY
Listen at volume. Then ask where AI genuinely helps
Turn hundreds of messages into a few clear insights, then decide honestly whether AI belongs in your product, and test it by hand.
Research at volumeTag, count, quoteWhere AI belongsOpportunity CardWizard of Oz
Present slides ▶
Full screen, one slide at a time · 12 slides
By the end of tonight you can…
- Combine sources (events, support messages, reviews, interviews) and say what each one misses
- Tag feedback at volume into themes with counts and exact quotes, and audit a sample of the tagging
- Judge where AI belongs in a product and where a rule, a person or nothing is better
- Describe one AI opportunity: job, input, judgement, output and the cost of a mistake
- Run the job by hand with Claude on ten real, anonymised inputs before writing any code
OUTPUT · Insight report + AI Opportunity Card (with ten hand runs)
Sources at scale: each sees part of the picture
- EVENTSWhat people did, at scale. Shows where; never says why
- SUPPORT MESSAGESWhat went wrong, in their words. Skews to problems and to people who write in
- REVIEWSPublic, unprompted opinions. Skews to the delighted and the furious
- INTERVIEWSDepth and context on past behaviour. Few people, so never counted as a share
- SALES CALLSWhy buyers hesitate or say yes. Skews to people who were already interested
No single source is the truth. A theme you hear in two sources and see in the events is worth acting on.
Feedback to insight at volume
- COLLECTOne file, one row per item: ID, source, date, text. Personal details removed
- TAGA short list of themes, each with a one-line definition; one main tag per item
- COUNTItems per theme, with the base: “38 of 200”. Note which sources they came from
- QUOTEExact words with the item ID, so anyone can find the original
- LINK TO BEHAVIOURConnect each theme to what people did: a leak, a cohort, an event count
An insight is a theme, a count, a quote and a behaviour. Any one of them alone is just a note.
Tag 200 feedback items, then audit a sample
I’m researching why people stay with or leave [MY PRODUCT]. Attached are [NUMBER] feedback items, one per row with an ID, a source and the text, with personal details removed. I want to find the most common problems so I can choose what to work on next. <feedback>[ATTACH FILE]</feedback> Propose 5–8 themes, each with a one-line definition, then give every item one main theme. Show a table: theme, count out of [NUMBER], and two quotes copied word for word with their item IDs. Tag an item “unclear” if you’re unsure rather than forcing a fit. Finally, list 20 item IDs chosen at random with the tag you gave each, so I can check your tagging by hand. Done means the table, the definitions and the 20-item audit list.
Check the response
- Counts add up to the total, with “unclear” shown
- Each quote is found word for word in the file
- You agree with most of the 20 audited tags
Where AI belongs, and where it doesn’t
Good fits
- Judgement on messy text that no simple rule can handle
- Drafting and summarising, for a person to edit before it goes out
- Classifying and extracting: tag a message, pull out days and times
- Mistakes are cheap, visible and easy for a person to correct
Poor fits
- Anything a simple rule does reliably: totals, dates, “if paid, send receipt”
- Moving money: charging, refunding or paying out
- Irreversible or legal decisions with no person checking
- Nobody can tell whether the answer is right, and nobody checks
Use AI where judgement on messy input is the hard part and a person stays in control. Otherwise, don’t.
The AI Opportunity Card
- JOBThe task for the user, in one sentence, tied to an opportunity on your tree
- INPUTWhat the AI would read: real examples, how messy, and any personal data
- JUDGEMENTThe hard part a rule can’t do. If a rule can, write the rule instead
- OUTPUTWhat it hands back, who sees it, and what they do with it
- COST OF A MISTAKEWhat happens if it’s wrong, who notices, and how easily it is undone
If you can’t name the judgement, or a mistake is costly and unseen, the honest answer is “no AI here”.
From a parent’s rambling message to a proposed slot
Starting point
“Let’s add AI to TutorDesk: a chatbot that handles all the bookings for tutors.”
Reframed
Job: turn a parent’s messy message into one proposed slot. Input: the message + the tutor’s free slots. Output: a proposal the tutor approves or edits. Mistake: cheap; the tutor decides.
Ask
- What is the judgement here that a rule couldn’t do?
- What should happen if the message names no day at all?
- Why is “the tutor approves” essential to the cost of a mistake?
- Which opportunity on Tuesday’s tree does this serve?
AI inside a feature vs AI as the whole product
AI inside a feature
- One job inside a flow that already works: the slot proposal
- If AI fails, the product still works: the tutor picks a slot
- Value comes from your workflow, users and data, not the model alone
- Easier to test: one input, one output, one person checks
AI as the whole product
- The AI’s answer is what people pay for
- If it is wrong or slow, the product fails
- Harder to stand out: others can build on the same models
- Needs strong evidence of quality from day one
In loop 2, start inside a feature: one job, a fallback, and a person in control.
Do the job by hand with Claude before any code
I’m testing whether an AI feature in [MY PRODUCT] could do this job well, so that I can decide whether to build it at all: [JOB FROM YOUR CARD]. Below is one real input from a user, with names and contact details removed, and the context the feature would have. <input>[PASTE ANONYMISED INPUT]</input> <context>[PASTE CONTEXT, E.G. FREE SLOTS]</context> Produce exactly what the feature would show the user: [OUTPUT, E.G. ONE PROPOSED SLOT AND A ONE-LINE REASON]. If the input is ambiguous or missing something you need, say what is missing and the question you would ask, instead of guessing. I’ll compare your answer with what I would have done, so keep it short. Done means the output and, separately, anything you were unsure about.
Check the response
- Inputs are real and anonymised, not invented
- Each run is marked right, edited, wrong or asked
- Ambiguous inputs produce a question, not a guess
Insight report, AI Opportunity Card, ten hand runs
Turn your feedback into insights, then test honestly whether AI can do one useful job for your users.
- Tag your feedback with Claude; audit 20 items by hand and record how many you agree with (14 min)
- Write the insight report: top three themes, each with a count, quotes and a behaviour (12 min)
- Fill the AI Opportunity Card for one job, or show why a rule or a person does it better (10 min)
- Run ten real, anonymised inputs in Claude.ai; mark each right, edited, wrong or asked (15 min)
- Decide: build it into the product, or “no AI here, because…” plus your internal tool (4 min)
Deliverable
Insight report + AI Opportunity Card in product/07-ai-opportunity.md
Insight and AI opportunity rubric
- The report has 3+ themes, each with a count and base and a verbatim quote with its item ID
- The tagging audit states agreement out of 20 sampled items and what was corrected
- Each theme links to a behaviour: an event count, a retention cell or a leak
- The card names job, input, judgement, output and cost of a mistake for one opportunity
- Ten real, anonymised inputs were run by hand, each marked right, edited, wrong or asked
- It ends in a decision: build it, or “no AI here, because…” with the internal tool named
NEXT · Week 8: the business model and the AI bet. Bring your card, ten runs and every “wrong”.
© 2026 MVP Studios Ltd. MVPAI™ and the MVP Founders Programme materials are original works of MVP Studios Ltd. All rights reserved. No copying, redistribution or commercial use without written permission. Course terms