User Interviews for Product Managers

I’ve conducted user interviews across all stages of the product. I like to define user interviews in 3 distinct types: Problem space, Solution space, the Usability/Experience Test. In the post below I talk about each type and provide some guiding questions.

Stage 1: Problem Space

Goal

The goal of interviews in this stage is to understand your potential users’ problems and how they’re solving those problems to validate/invalidate your hypotheses

Timing

In the ideation phase before launching a new product or feature , you should have a hypothesis of what problem you’re trying to solve and a vague idea of a potential user base.

Execution

Whom to Interview

Finding users can be a bit of a challenge in this stage, the following approaches have worked for me:

  1. Posting in your networks for the target user base or ads on Craigslist/Social platforms
  2. Consulting with a user research firm if possible for your company ( Ex. https://www.answerlab.com/)
  3. Use a tool like https://www.userinterviews.com/ to find users you want to target

You can offer users gift cards, enter them in a lottery or provide early access. Tools like User Interviews will take care of payments as well.

Pre-Interview Prep
  1. Once you have interviewees, make sure you have hypotheses written down about your product and user assumptions.
    • If you need a starting point, it can be helpful to layout a “Jobs to be Done” framework for your potential users to help with the hypotheses
  2. Helpful questions at this stage probe into understanding your users. Some examples to use as a framework are:
    • Walk me through a typical day in your life (for consumer) , walk me through steps of your job (for SaaS)
    • How would you want to solve [problem your product will solve] (for consumer)? What would success in your role look like? (for SaaS)
    • How are you solving the problem now?
    • What are the biggest pain points in solving that problem?
    • Can you rank the above pain points?
    • What other tools/apps/software are you using?
    • How do you discover the [some tools from above]?
    • What do you like/dislike about [tools from above]?
    • Ask questions related to your hypotheses but make then unbiased if you can’t get those answers in the above questions.

It’s crucial to stay open minded to other ideas/hypotheses that may come out of interviews that you may not expect.

Take some time and delve into the above questions, especially the pain points. Repeat the process for as many users you need for validation (around 10 is usually a good sweet spot)

During and post interview
  1. Make sure users know there are no right or wrong answers
  2. Don’t judge the answers of the users
  3. Dig deeper into anything surprising
  4. Go back to your hypotheses and see if you can validate/invalidate with each user’s interview
  5. Distill the learnings and share with your team! It’s important to involve designers, engineers and rest of the team in this process
    • One good way to distill some learnings are to build user personas
  6. If it’s users from your network/someone you can stay in touch with, keep them in your network or build a small community to keep these users around for future iterations

Stage 2: Solution Phase

Goal

The goal of interviews in this stage is to understand if the solution you and your team came up with for the problem is what users would expect/use?

Timing

The testing is usually done with prototypes or some mock-ups of the product/feature you are building.

Execution

Whom to Interview
  1. E-mail existing users if you have an MVP product
  2. If there’s a way to surface a form inside your product, you can ask users to sign up there if it’s a new feature
  3. Any users you retained from the ideation phase would be great to test out solutions if you don’t have any users yet
  4. For your last resort, you can find users the same way you did for ideation phase with agencies/tools

You can offer users gift cards, enter them in a lottery or provide early access.

Try to get different user personas you have identified for your product to get a diverse perspective.

Pre-Interview Prep
  1. Once you have interviewees, make sure you have hypotheses written down about your product and user assumptions. Hypotheses will be key to every user interview
  2. Have a script ready for walking through the product with the interviewees
  3. Send users links to have access to the product before the interview
  4. Helpful questions at this stage probe into understanding your users. Some examples to use as a framework are:
    • [Have users walk through prototype] What are your initial thoughts looking at this product?
    • What do you think you should do here [while showing each screen]?
    • How do you feel about this [showing any features you have doubts about]?
    • What do you like/dislike about this[on some of the bigger features/screens]?

Take some time and delve into the above questions, especially the dislikes. Repeat the process for as many users you need for validation (around 6-7 is usually a good sweet spot)

During and post interview
  1. Make sure users know there are no right or wrong answers
  2. Don’t judge the answers of the users
  3. Go back to your hypotheses and see if you can validate/invalidate with each user’s interview
  4. Note any design confusions for the product
  5. Distill the learnings and share with your team! It’s important to involve designers, engineers and rest of the team in this process

Stage 3: Usability/Experience Testing

Goal

The goal of interviews in this stage of a product can be several different things, including:

  1. How to improve the product?
  2. What do users love most about your product?
  3. What are UI improvements you can make to the product?
  4. How intuitive and easy-to-use is the product?
  5. What messaging resonates with your users as the value prop? This could also inform marketing

Timing

The testing is usually done with an MVP product you built with your solution research and usually helps with refining and tweaking the product/feature.

Execution

Whom to Interview
  1. Existing users if you have an MVP product
  2. Any users you retained from the ideation phase
  3. Find users the same way you did for ideation phase with agencies/tools

You can offer users gift cards, enter them in a lottery or provide product benefits!

Try to get different user personas you have identified for your product to get a diverse perspective.

Pre-Interview Prep
  1. Once you have interviewees, make sure you have hypotheses written down about your product and user assumptions. Hypotheses will be key to every user interview
  2. Have a script ready for walking through the product with the interviewees
  3. Send users links to have access to the product before the interview
  4. Helpful questions at this stage probe into understanding your users. Some examples to use as a framework are:
    • [Have users walk through prototype] What are your initial thoughts looking at this product?
    • What do you think you should do here [while showing each screen]?
    • How do you feel about this [showing any features you have doubts about]?
    • What do you like/dislike about this[on some of the bigger features/screens]?

Take some time and delve into the above questions, especially the dislikes. Repeat the process for as many users you need for validation (around 6-7 is usually a good sweet spot)

During and post interview
  1. Make sure users know there are no right or wrong answers
  2. Don’t judge the answers of the users
  3. Go back to your hypotheses and see if you can validate/invalidate with each user’s interview
  4. Note any design confusions for the product
  5. Distill the learnings and share with your team! It’s important to involve designers, engineers and rest of the team in this process

Experiment Readout Template

Template: Reporting experiment or project results

Keeping track of the performance of an experiment or feature that you shipped, either in emails or an evolving document, is essential for maintaining stakeholder alignment and boosting your visibility as a product manager. This template is a guide for reporting on a completed project to colleagues that have varying degrees of familiarity with your work.

Subject / Title

[Project Name – Date]

A descriptive email subject or document title will help your teammates quickly find your documentation. Describe your project and include relevant dates.

Examples:

  • Home Page Copy Experiment – 8/21/2020 update
  • [Android] Onboarding Funnel Optimizations (8/1 – 9/7)

TL;DR

The goal of this [project/experiment] was to test [X]. We learned [Y]. Next, we plan to do [Z].

Include a 2 – 3 sentence summary of your goal (project objective or hypotheses tested in an experiment) along with the high-level results. This is the first thing a reader will see. It should give readers a quick insight into your project, and those who are interesting in learning more can read further. This section is essentially an executive summary.

Context

  • Background info 1
  • Background info 2
  • etc.

Answer the following questions in a short paragraph or a few bullet points:

  • What user or business problem were you trying to solve?
  • Why were you trying to solve this problem? i.e. how do you know this problem is real?
  • What was your solution? Describe the feature, project, or experiment as you would to a friend who has never heard about it before.

Include screenshots where helpful.

Experiment Setup

Experiment name

Start Date – End Date

  • Variant 1 (allocation %): Description, hypothesis
  • Variant 2 (allocation %): Description, hypothesis
  • etc.

If you’re reporting on an A/B test, this section is useful for detailing the logistics of your experiment. Think about what your audience would need to know to understand the results you are reporting.

For example:

  • Experiment duration, start and end dates
  • Number of variations you tested vs the control group
  • Allocation in each variant
  • Differences in the treatments of each variant
  • Hypotheses being tested

Results

Metric 1 moved because of X. Metric 2 moved because of Y.

Variant 1Variant 2
Metric 1x% (stat sig status)x% (stat sig status)
Metric 2x% (stat sig status)x% (stat sig status)

Report the results of important metrics, like your core product KPIs or predefined success indicators. With the data provided in this section, readers should understand the reasoning behind your decisions in the Next Steps section. Save any deeper analysis for the end of the email.

For experiments, it is important to indicate the percentage lift vs your control group and the statistical significance of each metric.

Example:

Variant 1Variant 2
ARPDAU-1.2% (stat sig)+0.7% (not stat sig)
Conversion– 3.0% (stat sig)-0.1% (not stat sig)

Next Steps

  • Plan for this project going forward – roll out to more users, end, iterate, etc.
  • Ideas for using the insights in other projects

What are you planning to do next and why? Use bullet points to talk about how you will use what you learned from this project to improve your product going forward.

Further Analysis

This optional section is where you can include interesting analysis or insights for readers that are looking for more details on your project, such as:

  • Funnel analyses
  • User interaction patterns – how are people using what you shipped?
  • Trends over time
  • Trends by different cohorts

Shoutouts

Huge thank you to X and Y for their great work on making this feature possible!

It’s always nice to recognize your colleagues for their contributions! If applicable, this is a great opportunity to recognize your awesome teammates for their hard work on the project you talked about in this email or document.

Short and Long Term Retention Benchmarks

I’m launching a consumer mobile app in a non-traditional category, it lies somewhere between design, mindfulness and games. I wanted to build a set of short and long term retention benchmarks for the product which sent me on a long hunt for retention numbers for mobile apps. I’m coalescing them in this post for anyone to use [with sources included].

Short Term (D1, D7, D30) Retention

Average across several mobile app categories: Adjust has published their 2020 report with retention curves averaged across categories.

Interact with the numbers in the original source: https://www.adjust.com/resources/downloads/app-trends-2020/?download=%2Fresources%2Fdownloads%2Fapp-trends-2020

Top 1000 Apps: Averages miss the distribution of retention numbers, so below is the TOP 1000 apps and their benchmarks, however this data is from 2018 but a good enough proxy:

Source: https://www.emarketer.com/chart/228505/mobile-app-performance-metrics-worldwide-average-30-day-retention-rates-by-category-2018

It’s interesting to note the difference between averages and the top 1000 apps. Ex. Casual Games D30 across all apps is 6%, however the top apps are around 12%

Specific Apps: I found a source with some specific apps skewed towards games:

Source:https://www.sequoiacap.com/article/retention

Hearsay: Retention numbers I’ve heard around the industry/seen on social media.

  • For Games: D1 retention 25% (top15% >35%), D7 retention 5% (top15% > 14%), D30 retention 1.5% (top15% > 6%)
  • Top social media app retention > Top Dating apps retention > Top Games retention > Top mindfulness/health apps

Long Term Retention (6 month, 12 month)

There is a great post that talks about these benchmarks (https://www.lennyrachitsky.com/p/what-is-good-retention-issue-29). I won’t repost them here.

Other types of Retention

There’s other retention metrics like (Week 1, Week 4, Month 1 etc) that are also helpful for benchmarking but I haven’t found too many good benchmarks around here. However, some back of the napkin math from D1,7,30 and combination with DAU/WAU can help get close to the numbers. Here is a helpful chart (from 2016 data):

source: https://www.quora.com/What-are-typical-long-term-retention-rates-for-a-consumer-utility-app-I-am-trying-to-understand-what-the-bad-good-and-crushing-it-numbers-would-be/answer/Tasmin-Singh

Other Resources:

https://mixpanel.com/topics/what-is-a-good-app-retention-rate/

https://www.geckoboard.com/best-practice/kpi-examples/retention-rate/

I hope these are helpful! If you have any retention numbers to share, please reach out and I’d love to add them here. Find me on Linkedin/Twitter!

First PM and First-time PM here. What should I do in the first 30/60/90 days?

This is a blog post I had written in response to Ask Women in Product series to the question: “ I am the first PM hire at a B2C startup, and I am also a first-time PM. What should I focus on in my first 30 / 60 / 90 days?

Being the first Product Management hire when it’s your first time in the Product Manager role is an incredibly rewarding experience. It’s a rare growth opportunity that many PMs seek out only in the later stages of their career.

The journey starts with the PM being a student of the product and continues as you grow into a subject matter expert. I find it helpful to break down the first 90 days into the different phases of the PM’s knowledge growth.

First 30 days: the Learning Phase

Spend the first 30 days learning everything you can about your users, your product and market, your company, and the tools that your teams use.

Users

To do the job of a PM well, you must first understand your users. Learn about the user segments that use your product. In B2C products especially, you’ll want to leverage analytics to understand your users’ behavior.

Ask the team about the (qualitative and quantitative) trends they’ve seen so far, then take the time to do a deep dive into the data. You’ll want to ask questions like:

  • What are the key user cohorts of the product?
  • What is the set of power users? What is their typical behavior?
  • What are the core problems faced by each user segment?

If there isn’t an extensive data infrastructure that tracks this data, ask the team for the consumer insights that they’ve learned so far. If there’s no information available, see if you can run a survey or a focus group with some users in various user segments and learn about them.

The Product and Your Market

Learn the industry and the market of your product. Look at the top competitors and become an expert in their products. Use your product with the aim of becoming an expert in it; learn its ins and outs. Study the competing products regularly to understand the differences.

Some questions to ask yourself:

  • What pain points do you encounter when you use your product? What about competing products?
  • What does your competitor’s marketing collateral imply about their user cohorts? Their value proposition?
  • What does your product lack that you see in your competitors’ products?

Find industry-specific reports that offer an outsider’s perspective and which provide data on the key drivers and differentiators in your market.

If you work in a regulated industry, you’ll also want to understand the rules that your company and products have to abide by.

The Company

Learn about the company’s goals, its mission, its founders, and meet the team. Here are a few core activities to get started:

  • Set up a 1:1 meeting with each team member and ask about their roles to learn the culture of each of the various disciplines;
  • Find out how the PM role was previously distributed throughout the team, then ask your teammates to share their expectations of the role that you now have. Knowing what people expect of you as the first PM will help you work well with other disciplines like engineering, design, and research;
  • Get to know the foundersthe company-level goals, and the high-level strategy for the product(s). Learn about the mission and main goal of your company in addition to understanding your manager’s expectations.

The Tools

Take some time in the first 30 days to learn the tools. You’ll want to understand any analytics infrastructure as well as its limitations. Make notes of what is missing, if anything, and what you’d like to add in the future. If there is no analytics infrastructure, ask your team how they have been tracking main user events and add analytics to your backlog. If needed, take some time to define the metrics strategy for your product. A strong product analytics infrastructure becomes more crucial as you scale.

You’ll also want to learn what technical tools the engineers are using. This knowledge will help you understand your product’s technical design at a high level going forward.

Aside from learning the analytics stack and becoming familiar with the engineering tools, you may need to introduce product management tools (if the company has not been using them yet). Some of my favorite product management tools include:

  • Amplitude for product analytics and event tracking;
  • Trello for organizing and tracking tasks;
  • Slack for team communication; and
  • Invision for design and collaboration.

First 60 Days: the Growth Phase

In the next 30 days, your focus will shift from knowing the background to learning how to grow the product and yourself. Let’s break this phase down into three parts:

Personal Career Development

Since this is your first PM role, you will want to learn from other PMs about best practices. Leverage these and similar communities to build a network of experienced PMs who can be your sounding board. With luck you may even find a mentor:

Make every effort to hone your product intuition. Also, get an in-depth understanding of the tasks of a PM and assess your strengths and weaknesses. For example, you might be good at analytics but lack strategic skills. Once you identify your weaknesses, work on building those specific skills.

Here are some helpful resources that you can use to acquire new skills:

Product Development

Now that you understand and know your product and its user base, use that knowledge to figure out how to make the product better to achieve your product goals. You can make the product better through several approaches:

  • New feature development. Ideate and specify features that add significant value to your product and enhance existing features; and
  • Optimizations. Optimize the existing features of a product or an existing user experience;
  • Technical improvements. Introduce technical improvements, which for B2C products, often lead to better user experiences. For example, there may be room to improve error rates, load times, or fix some tech debt.

As the only PM, you will have to figure out not only what to build but also when to build it. Align with the founders and stakeholders on your short and long-term roadmap so everyone has a common understanding of priorities.

Note that one roadmap template will not fit all products nor all companies. Don’t be afraid to use frameworks from various resources as long as you adapt them to the needs of your company and stakeholders.

Team Building

In this second phase, you should spend time building a good relationship with your team. As the first and only PM, you need to gain your team’s trust and assure them that you are capable of making good, informed decisions. Continue the 1:1’s with your teammates and discuss your ideas for the product openly so you can validate your product thinking.

Be open to other stakeholders who may have ideas and ensure that everyone feels like they are a part of the roadmap process. Be open to ideas but as a PM, it is also your responsibility to ensure the expectations are realistic. Handle unrealistic requests by explaining why they aren’t possible and support your explanations with data.

First 90 Days: More Growth + Execution

The third phase — Days 61 to 90 — is a continuation of the learning and growth that were the focus of your first two phases. In this phase, however, we add one key component: Execution.

Execution

For the first 60 days, you’ve been learning the basics of the market and the product, and you’ve been working on growing your PM skills. It’s time to apply that learning and start executing on your ideas and goals.

Work with your team to ship a small feature or optimization and figure out the process that works best for your team.

As the first PM, you will set the precedent for how a specification document should be written, who will manage the shipping deadlines, and how smoothly the process works. Once a product has shipped, evaluate its performance and communicate the results to the team. If you were not successful, determine what needs to change. These activities are all part of the product delivery process and execution that you will constantly be doing as a product manager.

In closing

Being the first and only PM at a B2C startup is a great opportunity to very quickly learn and grow as a product manager. The changes that you make at an early stage often yield results quickly and serve as great learning opportunities. I hope the ideas that I’ve shared in this piece prove to be helpful.

Further Reading

Here are some other resources on Product management:


Thank you to @mdy for editing this piece.

How do I get promoted to Senior PM?

This is a blog post I had written in response to Ask Women in Product series to the question: “I really want to be promoted from PM to Senior PM. What do you think distinguishes these positions, and how can I best show my achievements to make my case?

The transition from a Product Manager (PM) to a Senior Product Manager role encompasses growth in all the skills required to be a product leader.

You’ll want to start by making your intentions clear to your manager and setting goals for how you can improve to get promoted to the next level. Below, we explore the various skills that a PM will need to level up to get promoted to a senior position.

Strategy

The term “Strategy” consists of roadmap strategy, individual feature strategy, as well as long- and short-term product vision. The difference between a product manager and a senior product manager in terms of strategy is as follows:

Image for post
Strategy for PMs and Senior PMs

To level up: Ask your manager for ownership of a big product goal. For example, ask to own the revenue growth of your product and create a roadmap to meet the product goals. Use the project both as a learning experience and as a way to showcase your ability to strategize. Work on gradually owning larger parts of the product roadmap.

Analytics

The term “Analytics” refers to a combination of technical analytical skills, the ability to identify key metrics, derive business insights, and experiment. The differences between PM and senior PM are as follows:

Image for post
Analytics for PMs and Senior PMs

To level up: Derive key insights from your product analyses and make recommendations that contribute to the business as a whole. Keep focusing on your product but also drive initiatives that impact larger product goals.

Product Design/Product Sense

“Product design and product sense” are terms that refer to both how well a PM understands users and their needs as well as the PM’s ability to design products to meet those needs.

Image for post
Product Design or Product Sense for PMs and Senior PMs

To level up: Deconstruct top products in your market and do market research. Use your research findings to make recommendations to your product team. Take the initiative to share your research and learnings with the broader organization.

Execution

The term “Execution” refers to the PM’s ability to drive product strategy and roadmaps with multi-disciplinary teams, thus ensuring that ideas go to production.

Image for post
Execution for PMs and Senior PMs

To level up: Build great relationships with other disciplines by setting up 1:1s and understanding their needs. Use your insights to create better processes so everyone can execute more efficiently.

Next Steps to Level Up

The expectations from a senior PM are company-dependent, so the very next step to working toward a promotion should be to align with your manager and company on the expectations. Evaluate your areas of strength and weakness in the skills expected of a Senior PM and work from there.