Feedback from 3 Different Chatbots
I am currently working on a product sense grading prompt. It is more work than you might imagine to get it right. For the results to be genuinely helpful for me and others.
In my newsletter this week I shared the audio of my answering the prompt: Product to enjoy art. Here are how the three different chatbots responded.
One of them got things wrong, they missed things that were in the transcript. Not unlike a real panel, you will see a high, medium and low response. You see overlapping feedback but also where things diverge and people have different opinions on what the output indicated.
I am debating about continuing to use all three to get different POVs but it is likely too much work to keep up after testing is complete.
If you listened and want to compare your results to the prompt that is slowly being programmed to be my doppleganger.
ChatGPT
Executive Summary
Strong, structured product sense with clear prioritization. You framed the mission (“help people enjoy art”), narrowed ambitiously yet pragmatically to novices in the online context, prioritized pain points via frequency × severity, and proposed three distinct solutions, then prioritized with Addresses Need × Engineering Effort and gave a workable verbal flow for the top pick. Biggest gaps: make the “why now” tie back to company strategy, state explicit risks & tradeoffs, and offer goal-aligned metrics. Overall, this reads like a solid Hire with a few polish items away from “Strong Hire.”
Scores
Overall Recommendation: Hire
Gate-Acknowledged Score: 4.3 / 5
Adjusted Potential Score (if gates passed): N/A
What You Did Well
Clear mission + scope: “Help people find enjoyment in art,” then focus on novices and online context.
Prioritization discipline: Used TAM/Opportunity to choose online; frequency × severity to select “articulation” as the top pain; Need × Effort to pick the phone app.
Solution breadth → depth: Three distinct concepts (flip book, translator, phone app) with a concise verbal flow for the phone app.
Audience-aware check-ins: Frequent “pause/any questions before I prioritize?” reduced cognitive load.
Where To Improve Most
Make “why now” strategic: You cited pandemic/AI shifts; explicitly connect to company right-to-win (distribution, data, partnerships) and user adoption levers.
State explicit risks & tradeoffs: (e.g., privacy from photo ingestion, artist IP/licensing, on-device vs cloud inference costs) and how you’d mitigate.
Define success with metrics: Tie to mission (enjoyment → discovery/usage outcomes), not generic engagement.
Sharpen two-sided view: You noted artists/museums; briefly justify sequencing (start with consumers; later unlock creator marketplace).
Clarifying Questions & Assumptions
Scope of “art”: Confirmed 2D/3D visual (paintings, sculpture).
Company context: Assumed startup; later sell to museums or stand-alone
Constraint posture: Assumed no hard budget to explore solution space.
User focus: Chose novice over expert to maximize TAM and utility.
Strategic Setup
Mission & goal: Help people enjoy art; refined to “help people find the art they want to enjoy by improving articulation.”
Why now (good but thin): Pandemic-driven digital consumption + AI/ownership debates; strengthen by tying to distribution advantages and creator economics.
User Segmentation
Primary split: Novice vs. expert; chose novice intentionally.
Context segmentation: Online vs. museum vs. home; prioritized online using TAM and frequency of exposure/opportunity.
Pick rationale: Online offers highest reach and under-served “enjoyment” mechanics beyond mere exposure.
Pain Points
List (focused): Articulation/definition, searching, overexposure/habituation for screensavers/backgrounds.
Prioritization: Selected articulation using frequency × severity (search depends on articulation; overexposure affects fewer users).
Solutions
Set: (1) Flip book style explorer → vocabulary; (2) Translator chat tool from vague descriptors to style; (3) Phone-camera “I like” collector → weekly recs.
Prioritization: Addresses Need × Engineering Effort → pick phone app (habit-piggyback, lighter MVP).
Verbal flow present: Download app → capture liked visuals/screens → model infers style → periodic suggestions (screensavers/backgrounds) → optional artist marketplace later.
Add next time: 1–2 explicit risks (privacy/IP, cold start, model bias) + mitigations; outline MVP vs V2 (on-device embeddings now, creator marketplace later).
Metrics
If unasked, metrics are optional; here are aligned examples you could state next time:
% users who save ≥1 recommended artwork/week (activation).
Time-to-first “I like” recommendation ≤7 days (onboarding success).
Weekly “kept as background ≥7 days” rate (enjoyment persistence).
Creator payout rate / $ per active creator / week (when marketplace launches).
Recommended Practice
Keep the check-ins; add a 10-second gate after each major section (“pause/adjust?”).
Always name 1–2 concrete risks per top solution with mitigations.
Anchor “why now” to right-to-win (data, channels, ecosystem).
Suggested Reading
Mock Count Recommendation
10 mocks (aim for all ≥4). Focus reps on: explicit risk/tradeoff articulation, crisp “why now” with strategic tie-in, and rapid metric framing.
Claude
Product Sense Interview Evaluation
Executive Summary
The candidate demonstrated solid product thinking fundamentals with clear structure and logical flow. They effectively clarified the problem space, established a user-focused goal, and provided creative solutions with good prioritization frameworks. However, critical gaps in explicit prioritization criteria and missing solution details prevented higher scores. The candidate showed strong communication skills and adapted well to interviewer questions, but needs to be more precise with frameworks and include technical implementation details.
Scores
Overall Recommendation: Leaning Hire
Gate-Acknowledged Score: 3/5
Adjusted Potential Score (if gates passed): 4/5
What You Did Well
Strong clarifying questions - Asked fundamental questions about art scope, company context, and budget constraints before diving in
Clear goal articulation - Established "help people find the art they want to enjoy" as a user-focused mission
Creative solution generation - Proposed three distinct solutions (flip book, translator tool, camera app) with different approaches
Good communication structure - Used clear transitions and logical sequencing throughout the interview
Thoughtful user journey mapping - Walked through specific pain points like screensaver selection and overexposure
Where To Improve Most
Missing explicit prioritization criteria - Used vague terms like "opportunity" and "need" without defining concrete frameworks (this capped your score)
Insufficient solution depth - Lacked verbal wireframe/flow description for chosen solution with specific screens and user actions
No risk/tradeoff discussion - Failed to identify implementation risks or potential downsides of proposed solutions
Imprecise segmentation framework - "TAM and opportunity" criteria were not clearly defined or measured
Clarifying Questions & Assumptions
Effective fundamentals coverage - Started with core problem definition (art scope) rather than tactical details
Smart assumption validation - Confirmed startup context and budget flexibility appropriately
Could have probed deeper - Might have asked about target demographics or existing competitor landscape
Strategic Setup
Clear mission statement - "Help people find enjoyment in art" was well-articulated and user-focused
Solid "why now" reasoning - Connected pandemic digital shift and AI/creativity debates to current market timing
Good market insight - Recognized intimidation factor and accessibility barriers in art consumption
User Segmentation
Reasonable segment identification - Expert vs novice distinction made sense, then refined to online/museum/home contexts
Weak prioritization framework - "TAM and opportunity" criteria were mentioned but not clearly defined or measured
Logical segment choice - Online focus was sensible given accessibility and frequency arguments
Pain Points
Good quantity and specificity - Identified relevant pain points like articulation difficulty, search challenges, overexposure
Attempted prioritization - Used frequency × severity framework appropriately
Clear selection rationale - Chose articulation problem with sound reasoning about foundational nature
Solutions
Met requirement - Provided three distinct solutions with different approaches
Missing verbal wireframe - Lacked detailed flow description with specific screens, states, and user interactions
No explicit risks identified - Failed to discuss implementation challenges or potential failure modes
Good prioritization attempt - Used "addresses need × engineering effort" framework appropriately
Metrics
Not attempted - candidate focused appropriately on core problem-solving given time constraints.
Examples if needed:
Weekly active users creating style profiles
Percentage of users who save recommended art within 7 days
Average time to find desired art style (reduction metric)
User satisfaction score on art recommendations
Recommended Practice
Practice explicit framework articulation - Define all prioritization criteria with specific measurements
Develop solution wireframing skills - Practice describing user flows with screen-by-screen detail
Include risk assessment - Always identify 1-2 implementation risks and mitigation strategies
Suggested Reading
Mock Count Recommendation
10 additional mocks recommended to practice framework precision and solution depth.
Google Gemini
Executive Summary
The candidate delivered an exceptionally strong and well-structured performance, demonstrating a masterful command of product sense frameworks. They excelled at turning an ambiguous prompt into a clear plan, methodically moving from clarifying questions to a user-focused goal, needs-based segmentation, and prioritized pain points. Their use of explicit prioritization frameworks (TAM/Opportunity, Frequency/Severity, Need/Effort) in every major section was best-in-class. The primary area for improvement is pacing; the setup portion of the interview was lengthy, leaving less time for solution depth. While the proposed solutions were creative and well-reasoned, the final discussion lacked an explicit analysis of risks and tradeoffs for the chosen concept. Overall, this was a "Strong Hire" performance.
Scores
Overall Recommendation: Strong Hire Gate-Acknowledged Score: 5 / 5 Adjusted Potential Score: N/A
What You Did Well
Excellent Framework-Driven Approach: You began by outlining a clear structure and followed it precisely, making your thought process logical and easy to follow.
Consistent, Explicit Prioritization: You were one of the few candidates who correctly applied distinct, well-defined prioritization frameworks at every critical stage: User Segmentation (TAM × Opportunity), Pain Points (Frequency × Severity), and Solutions (Solves Need × Engineering Effort). This is a significant strength.
Strong User-Centricity: You established a clear, user-focused goal ("help people find enjoyment in art") and maintained that focus throughout, from identifying the "novice" user to defining their specific pain points.
Creative and Differentiated Solutions: You generated three distinct solution concepts (Flip book, Translator tool, Camera feature) that addressed the core user problem from different angles, demonstrating both creativity and strategic thinking.
Clear Communication and Check-Ins: Your communication was concise and structured. You effectively used pauses to gather your thoughts and checked in with the interviewer at key transition points.
Where To Improve Most
Pacing and Time Management: Your setup (from clarifying questions to pain points) was very thorough but took too long. You must move to the solutions phase more quickly to allow for greater depth, probing, and discussion of risks and GTM. Move to solutions by minute 20-22; use 10-second check-ins after each section to maintain momentum.
Explicit Risk & Tradeoff Analysis: For your chosen solution, you did not explicitly discuss the potential risks (e.g., cold start problem, privacy concerns with photo access, accuracy of the AI) or tradeoffs (e.g., prioritizing passive collection over active learning). This is a required component for a top-tier solutions section.
Conciseness in Setup: While your reasoning was clear, you can tighten your explanations during the strategy and segmentation phases to save time for the solutioning deep dive, which is the most critical part of the interview.
Clarifying Questions & Assumptions
Strong Start: You began perfectly by clarifying the ambiguous term "art" and establishing the scope (startup, no budget constraints).
Effective Framing: Your questions successfully reduced the problem space and demonstrated a focus on understanding the fundamentals before proceeding.
Strategic Setup
Clear Mission and Goal: You established a strong, user-centric goal upfront: "to help people find enjoyment in art," which you later refined effectively.
Compelling "Why Now": Your justification was excellent, linking the problem to current, relevant trends like the pandemic's impact on digital consumption and the rise of AI in creative fields.
Acknowledged Market Dynamics: You correctly identified the key players (creators, consumers, institutions) before strategically choosing to focus on the consumer side.
User Segmentation
Needs-Based, Not Demographic: You correctly avoided demographic traps and segmented your target "novice" user based on their context and job-to-be-done: enjoying art online, in a museum, or at home.
Excellent Prioritization: You used a clear framework (Total Addressable Market × Opportunity/Frequency) to justify your choice to focus on the "online" segment, which was well-reasoned and convincing.
Pain Points
Specific and User-Centric: You identified a strong list of 5+ pain points specific to the online novice, such as "defining what I like," "copyrights," and "overexposure."
Rigorous Prioritization: Your use of a Frequency × Severity matrix to select the top pain point ("defining and articulating what you like") was exemplary and demonstrated strong analytical thinking.
Solutions
Good Breadth and Creativity: You proposed three distinct and viable concepts that directly addressed the prioritized pain point.
Clear Verbal Wireframe: You provided a solid verbal walkthrough of your chosen "Camera phone" feature, describing the key user actions and outcomes (take pictures -> get style profile -> receive recommendations).
Missing Risk Analysis: As noted, the key missing piece was an explicit discussion of the risks and tradeoffs associated with your prioritized solution. For example, a risk for the camera app is the "cold start" problem (needing many photos to be useful) and a tradeoff is building a standalone app versus integrating into an existing photo platform.
Metrics
Not Discussed: This section was not covered, which is acceptable as you were not prompted for it.
Example Metrics: For your chosen solution, success could be measured by:
Activation: % of new users who add ≥10 photos to their style profile in the first week.
Engagement: # of recommendations saved or applied (e.g., as a screensaver) per user per month.
Goal Fulfillment: User satisfaction score (NPS or survey) on the question, "How well does this app help you discover art you enjoy?"
Recommended Practice
Time-Box Your Setup: Practice mock interviews with a timer. Force yourself to complete the clarifying, strategy, segmentation, and pain points sections within 20 minutes to ensure at least 15 minutes remain for solutions and probing.
Drill Risks & Tradeoffs: For every solution you propose in practice, create a habit of immediately listing 1-2 key risks and 1-2 key tradeoffs. This will make it a natural part of your flow.
Suggested Reading
Mock Count Recommendation
Based on this performance, you need 4 more mock interviews. (All 5 required mocks must score a 4 or 5).
