What Should I Test This Week If I'm New to AI Product Work?

Starting out in AI product management can feel overwhelming—between deciphering model outputs, defining success criteria, and figuring out what to ship first. I remember when I made the leap from traditional B2B SaaS PM to AI product lead; what helped me most was anchoring testing efforts around concrete user tasks, defining evals with real-world context, and using simple yet powerful tools to systematically measure progress.

In this post, I’ll share a pragmatic approach inspired by my experience (plus insights from innovators like PayPal, H2O.ai, and PM Toolkit). Whether you’ve got a freshly trained model or inherited a team building AI-assisted workflows, you’ll get actionable guidance on what to test this week to gain meaningful momentum.

Why "The Model Is Not the Product"

This phrase might sound like PM jargon, but it’s a guiding principle that saved me more than once. AI models are powerful components, but their raw outputs do not make a product. Instead, products solve specific user problems—often by automating or augmenting existing workflows.

For example, when PayPal started integrating AI into fraud detection, their focus was not to ship a "better model" but to reduce false positives that block legitimate payments (thereby improving user trust and retention). Similarly, H2O.ai emphasizes building AI solutions that support defined business metrics rather than chasing abstract model accuracy improvements.

So your first test should always be framed around a user’s task, not just model metrics.

Pick One Repetitive Task: Your North Star for Testing

AI’s biggest promise in product work is relieving humans from repetitive, time-consuming work—this could be drafting emails, categorizing support tickets, or summarizing meeting notes. Start by choosing one repetitive task that represents real value for your target users.

Example: If you’re building a content moderation tool, your task might be "Automatically flagging inappropriate comments."
Example: For a sales enablement AI, it might be "Drafting personalized follow-up emails."

This task becomes your testing anchor. It means any evaluation metric, user BLEU score meaning research, or prototype should map back to how well your AI supports that task.

Define an Eval Before Release: Three Questions to Guide You

Before you ship or test a new AI feature, define an evaluation framework. This prevents the common trap of “model wars” where teams argue about abstract improvements disconnected from real user impact.

I recommend asking yourself these three questions to define your eval:

What does success look like for the user completing this task? For example, "The user spends 30% less time triaging tickets" or "95% of flagged comments are truly inappropriate."
What failure modes are most critical to catch? AI behaves differently based on task context (task-dependent model behavior). For content moderation, missing a harmful comment is major; for email drafting, a mildly off-tone phrase may be acceptable.
How will you collect the necessary data to measure these outcomes? Can you log user interactions, collect manual labels, or run A/B tests?

Mapping evaluations to workflows helps keep model metrics like precision, recall, or F1 score grounded in product impact. Early on, spreadsheets are your best eval tool. Many teams start with lightweight eval sheets to compare model outputs side-by-side with human labels and user feedback.

The PM Toolkit calculators (which cover batch evals like sample size and A/B test planning) can help you quantify how much data you need to gather or how big a test you must run to get statistically sound results. Don’t ignore these tried-and-true measurement basics just because AI adds complexity.

Task-dependent Model Behavior and Failure Modes

One common mistake in AI product work is treating model performance as a single static metric. Great post to read In reality, your model’s behavior—and the consequences of its mistakes—depend heavily on the exact user task context.

For instance, when PayPal integrated AI for risk assessment, they noticed some transaction types required more conservative thresholds than others. Without accounting for this, you risk shipping a “one-size-fits-all” model that frustrates users in some workflows while underperforming in others.

Therefore:

Identify critical failure modes and their impact on user trust and business metrics.
Conduct targeted evals segmented by task variations.
Build lightweight monitoring or kill switches for high-risk areas (never ship without a kill switch!).

Tools from PM Toolkit like impact/effort prioritization and RICE scoring help prioritize which failure mode requires most urgent attention or iteration.

Automation vs Augmentation vs Innovation Patterns

To shape your testing strategy, it’s critical to know where your AI product sits along this spectrum:

Automation: AI fully replaces repetitive work, e.g. auto-approving low-risk transactions.
Augmentation: AI assists but the human stays in the loop, e.g. flagging suspicious cases for manual review.
Innovation: AI enables entirely new value, e.g. generating personalized financial advice.

Each pattern requires different test priorities:

Pattern Key Testing Focus Example Metrics Typical Evaluation Methods Automation Accuracy & low false positives Error rates, business impact Batch evals, A/B tests Augmentation User trust & efficiency Time saved, satisfaction User studies, qualitative feedback Innovation Adoption & discovery of value PMF score, NPS, conversion Pilot releases, iterative prototyping

For example, H2O.ai encourages teams to start with augmentation to build user confidence while collecting data for future automation opportunities. PayPal often blends patterns based on risk.

Choose the pattern most relevant to your project and tailor this week’s tests accordingly.

Next Step: Move Beyond Hypothesis to Measurable Outcomes

By now, you have:

Picked one repetitive task to focus your testing
Defined your evaluation around three guiding questions
Understood the criticality of task-dependent model behavior
Clarified if your AI solution is automation, augmentation, or innovation

Your next step is to translate this foundation into measurable outcomes by:

Setting up your eval spreadsheets to score model outputs according to user story examples, not just lab prompts.
Using PM Toolkit calculators to plan your sample sizes, A/B test designs, and expected impact on key metrics like retention or churn.
Building a simple dashboard to monitor failure modes and user feedback in real time after release.

Don’t fall into the trap of shipping without a kill switch or monitoring—remember that user trust is your most valuable asset when deploying AI.

Conclusion: Ground AI Testing In Real User Impact

Embarking on AI product work is exciting but can be daunting without a clear testing roadmap. The best advice I can give is to start small but test smart:

Pick a single repetitive user task where AI can add value
Define your evals with business and user outcomes top of mind—skip model wars
Keep task-dependent failure modes front and center and have kill switches ready
Align your testing approach with your automation/augmentation/innovation pattern
Use tools like PM Toolkit calculators and simple spreadsheets to measure, prioritize, and iterate deliberately

Companies like PayPal, H2O.ai, and PM Toolkit demonstrate that thoughtful AI product management is both an art and a discipline—grounded in customer understanding, data, and careful measurement.

Remember, building AI products isn’t just about better algorithms—it’s about solving real problems in ways users care about.

Happy testing!