Spec an AI essay-feedback feature and plan its launch
Frame the problem, test feasibility, write the PRD with an evaluation plan, and plan metrics and launch.
What you will have at the end
A problem statement, a one-day feasibility test, a PRD with behavior examples, a rubric and a launch bar, an unsure-case design, unit economics, a launch plan and a metrics plan. It is a complete AI product sample for interviews.
What you need
- Any chat AI tool
- A document editor
- A spreadsheet
All names, numbers and data in this lab are made up for practice. Never use real customer or personal data. Your ticks are saved in this browser.
The scenario
ClassNote is a fictional app used by 500 secondary-school teachers. Teachers say marking essay drafts takes hours, and feedback arrives late.
You are the AI product manager. The idea: an AI that drafts feedback on a student's essay for the teacher to edit and send.
Everything is fictional and no real student work is used. Make your own short sample essays for testing.
Starter material
Copy these into your own files or chat. Change anything you like.
Notes from three teachers. Use them to write the problem statement.
TEACHER A (history, 4 classes): Marking 30 essays takes me 5 hours at the weekend. I write the same comments again and again. I worry about being unfair when I am tired. TEACHER B (English, 5 classes): I want students to get feedback before they rewrite, not after. I would not let software grade. I would use a draft if I can change it. TEACHER C (science, 3 classes): I do not trust AI with students' names or grades. The school has strict privacy rules. If it is wrong, parents complain to me.
Three tiny sample student paragraphs for your feasibility test. Write ten more of your own.
1. "In my opinion the war was bad because many people died and it was sad." 2. "The treaty of Versailles was harsh. Germany lost land and had to pay. This made people angry, which helped the rise of extremists later." 3. "Photosynthesis is when plants eat sunlight to make glucose using carbon dioxide and water, and release oxygen."
Use this outline for the requirements document.
1 PROBLEM AND USERS 2 GOALS AND METRICS 3 SCOPE 4 BEHAVIOR EXAMPLES (10+) 5 QUALITY RUBRIC (with must-never-happen rules) 6 TEST SET AND LAUNCH BAR 7 WHEN UNSURE OR WRONG 8 DATA AND PRIVACY 9 SAFETY AND FAIRNESS 10 COST BUDGET 11 SPEED BUDGET 12 EXPERIENCE (screens and words) 13 LAUNCH PLAN 14 OPEN QUESTIONS
Fill in with current prices from your provider.
ESSAYS A MONTH PER TEACHER: 120 TEACHERS: 500 ESSAYS A MONTH: 60,000
TOKENS PER ESSAY (input + output, estimate): {{...}} PRICE PER 1,000 TOKENS: {{current price}}
MODEL COST PER ESSAY: {{...}} REVIEW AND SUPPORT COST PER ESSAY: {{...}}
COST PER ESSAY: {{...}} MONTHLY COST: {{...}}
VALUE: minutes saved per essay {{...}} x teacher hourly cost {{...}}
SENSITIVITY: usage doubles? A bigger model is needed? Heavy users?The steps
Do them in order. Each step says how you know it is done.
- 1
Write the problem statement
- Read the three interviews. Pick the problem to solve first, and say why.
- Write the problem statement with who, job, when, today, cost, success and evidence.
- Write what the problem is not about, for example grading.
Done when: A problem statement that uses evidence from at least two teachers.
- 2
Run a one-day feasibility test
- Write ten sample paragraphs of different quality, including one that is off-topic.
- Ask a chat tool for draft feedback on each. Score it with a simple rubric: accurate, helpful, kind, safe.
- Read every failure. Decide: promising, needs work or not ready.
Done when: A scored feasibility table and a decision with your reasoning.
- 3
Choose the automation level
- Use the levels from the course: suggest, draft, act with approval, act then review, full.
- Choose one for launch and write the evidence you would need to move up.
- Write what the AI must never do: for example give a grade, or send feedback without the teacher.
Done when: A chosen level and three must-never-do rules.
- 4
Write the PRD
- Fill in the outline. Write ten behavior examples with good and bad outputs.
- Write a rubric with three dimensions and the must-never-happen rules.
- Write the test set plan and the launch bar with numbers.
- Write what happens when the AI is unsure.
Done when: A PRD with all fourteen sections and numbers in the launch bar.
- 5
Design the experience
- Describe five states: first use, loading, a good draft, an unsure draft and an error.
- Write the exact words for each state and the next action.
- Design how the teacher edits, rejects and gives feedback on the draft.
Done when: Five states with exact words, and a clear edit and feedback flow.
- 6
Do the economics
- Fill in the unit economics with current prices. Work out cost per essay and monthly cost.
- Work out value per essay. Compute margin or break-even.
- Test sensitivity: usage doubles, or a bigger model is needed.
Done when: A cost model with sensitivity and a note on what you would cut first.
- 7
Plan the launch and the metrics
- Write the five launch stages with gates. Include a shadow stage with a few friendly teachers.
- Choose two metrics for each of use, quality, trust and value, with guard metrics.
- Write the stop rule and the weekly review agenda.
Done when: A launch plan with numeric gates, a metrics table and a stop rule.
- 8
Write the one-page summary
- Summarize: the problem, the decision on feasibility, the level, the bar, the cost and the plan.
- List the three biggest risks and who you would involve.
- Say what you would say no to in the first version.
Done when: A one-page summary that a leader could read in two minutes.
Success checklist
How to know the lab is really done
Show it in an interview
- Present the product in three minutes: problem, level, quality bar, cost and launch.
- Explain one thing you decided not to build, and why.
- Say how you would know in two weeks that the feature is working or not.
Common mistakes
- Letting the AI give grades
- Skipping the unsure-case design
- Ignoring cost until after launch
- Using only thumbs-up as a metric
Want a bigger challenge?
- Write the support script for teachers and a message to parents.
- Design the experiment that compares draft feedback with no feedback.
- Write a short privacy note in plain words for the school.
Next step: practice the interview questions for this role →
Other labs
A small agent that looks up orders and opens tickets, with checks, limits and a test set.
A researcher, a writer and a reviewer pass work to each other. You add limits and test the failures.
Play the forward deployed engineer for a fictional clinic: discover, scope, prototype, test and hand over.