Interview Prep Deep Dive - Product Execution
Interview Prep Deep Dive - Product Execution
What is it, the expectation, and how to win
This week, I’m going to return to the Interview Prep Deep Dive series and talk about Product Execution interviews. Feel free to revisit the last coverage on Product Sense, so you can compare and see the difference.
In short, product execution interviews evaluate your ability to make data driven decisions throughout the product life cycle. It includes how you connect the dots and set the right goals for the teams, how you prioritize and make trade offs, how you measure success, and how you root cause product problems.
Just like product sense, different companies might name it differently (e.g. product analytics), or it can also be blended into other types of interviews. Don’t get confused, just know that this type has the strongest focus on data, metrics, and analytical abilities.
Like product sense, I’ll take you through 3 parts:
🍭 Flavors of product execution questions
🔍 Interviewer assessments/expectations
📘 How to prepare
Ready? Lets dive in!
🍭 Flavors of product execution questions
There are 4 most frequently seen flavors:
Set goals and define metrics (for a specific product)
Evaluate a feature (the “why” and “whether”)
Make trade off decision
Troubleshoot
Set Goals and Define Metrics
“You’re the PM for <product XYZ | feature ABC>, how do you <set goals | define success> for your team? What are the north star / most important metrics you will track?”
The scope can be as big as “Instagram”, or as specific as “the like button on news feed”. Or anything in between. It can also be any type of product. Of course most likely it’s going to be software, but don’t get caught off guard if you’re asked to define success for your Tesla model 3 🙂. Approach them with the same structure which I’ll get to in a bit.
Evaluate a Feature
"How do you think <company A> decided to launch <product or feature XYZ>?”
e.g. “How do you think Facebook decided to launch “reactions” in the first place?”
“An engineer (or a user, stakeholder, etc.) comes to you with <a specific product idea>, how do you <respond | decide whether to pursue it>”
More tips to come but keep in mind, all these questions evaluate your data-driven decision making abilities (less about your strategy and behavior).
Make Trade-off Decision
“How do you decide between <competing features or options> to go with”
e.g. “How do you decide to place an Ads or Connection Recommendation modules on a given news feed slot?”
e.g. “How do you decide the best frequency of placing the ads on news feed - every 10 or 50 posts?”
e.g. “We’re giving away discount to incentivize conversion, how do you decide the best discount %? 5% or 15%?”
“(after running a test, or launching something) metric A is up, metric B is down, what do you do?”
You’ll be surprised with how different they appear to be, the fundamentals to tackle them are quite the same.
Troubleshoot
“You’re the PM for <product | feature XYZ>, you wake up and see metric A down by N%, what do you do?”
“<Something negative> happened to your product, what would you do?”
E.g. you discover that there are more Lyft drop offs than pick up in the airports. What would you do?
I’m sure you know, “call your data analyst” is not the answer interviewers are looking for. 🙂
—
Keep in mind, that the above types can be back to back parts of the same one big case question, or sometimes they can be separate.
Now lets look into interviewers expectations!
🔍 Interviewer assessment/expectations
To repeat, product execution interviews evaluate your ability to make data driven decisions throughout the product life-cycle.
Surprise to no one, product managers are expected to be analytical and data driven. It also means that product managers need to be deliberate about basing their decisions much more on facts than intuitions (e.g. to counter any bias or to not favor preferences of the highest pay grade). There are thousands of decisions need to be made from inception to post launch, so to ensure the best possible outcome for the product, it’s important to be able to come up with the best decision making framework feeding into any data that’s available or can be acquired.
Here are some specific areas the assessment would be focused on:
Communication and Structure: This is obviously a repeat, but again, a general but most critical assessment across all interview types. Can you listen well and communicate clearly and concisely? Are you structured in your thinking, communication, and how you approach problems throughout?
Big Picture Connection: also overlaps with product sense but it’s equally important here to always “start from the top” (aka what are the top level goals), and break it down from there, quantitatively.
Metrics Understanding: How well do you know several main categories of goals (e.g. growth, engagement, conversion, revenue etc.) a software company would usually care the most about? Do you understand how they tie to the goals, and how they relate to each other? How well do you understand the specific definitions of each types of metrics? Can you be very specific about how a given metric is defined or practically how they are captured or computed?
On the fly Framework: a bit different from just be structured in general, but specifically to your ability to create the right decision making framework given the situation at hand. Can you identify the right dimensions to create the framework based on what really matters (from the top)? Can you keep breaking things down, one layer at a time, unless you get what you need?
Again there are definitely some assessment overlaps with product sense, and these two interviews are not meant to be mutually exclusive after all. But to reiterate, product execution focuses much more on data-driven, quantitative decision making (vs strategy, user empathy, and creativity in a product sense interview).
—
So how do you prepare and tackle?
📘 How to Prepare
You may want to also revisit Land That Dream Product Job – Interviews for fundamentals.
For product execution, I’ll dive into each type of questions laid out above, followed by a number of reminders.
Answering Goals/Metrics Questions
High level, I’d usually approach in the following steps:
Clarify: make sure you clearly understand the question, the specific product in the question, and what specifically the interviewers are looking for. E.g.:
Make sure you’re aligned with the interviewer about even the obvious “Instagram Story” what it refers to and how it works
By “setting goals” - Confirm if the interviewer is looking for defining specific metrics or something else, and how deep and granular she’d like you to go
By “metrics” is the interviewer looking for one north star metric, or like top 3 metrics, or rather more comprehensive list
Pause: always, pause to organize your thoughts before proceeding.
Outline: outline your high level approach to set expectation and for interviewers to follow along (or to gain her early feedback whether this seems to be on the right track).
Start from Top: what’s the company’s mission and top business goals? Your don’t have to memorize the whole mission statement, and you don’t need to peek into any financial statement or listen to earnings call. There are some high levels you will know as a part of the general preparation when you kick off the process with a given company, and that’s sufficient. Don’t worry if it’s yet another company than the one you’re interviewing with. It’ll likely be a well-known company that you know the high levels of. Like Facebook, or Netflix. Lastly, when unsure, ask and discuss with the interviewer.
Connect: now discuss your specific product in scope, and how it connects to the top both in terms of mission and the business goals. Explain with clear rationale why this product exists, how it contributes to top level mission and business goals.
Define Metrics: now you’re ready define specific metrics using the connections we established above - what are some metrics that help you and the team measure our progress toward the contributions to the top?
Prioritize: which ones are the most important / representing to track, and why? What are the supporting metrics you also want to track in addition to north star
Avoid starting with a long laundry list of metrics. Approach it from high level down, one layer at a time.
Answering Feature Evaluation
High level steps:
Clarify: likewise, clarify to make sure you understand the questions, the feature in hand, and what the interviewer is looking for specifically before proceeding.
Pause: enough said.
Outline: enough said. Outline your high level next steps for interviewer feedback.
Data-Driven Problem Discovery:
start with asking yourself: “what specific problem might we be solving for with this?”
discuss how this problem might be discovered using data. E.g. look into user segments based on dimensions or user behaviors. What specific metrics might we be looking at
Determine why/whether this problem is worth solving based on what metrics would be impacted, and (again) how it connects to the top
Form Hypothesis on Solutions: once the problem is well defined and determined to be worth solving, going back to the feature to be evaluated and form a hypothesis on how it might (or might not) solve the specific problem
Test: The best way to confirm the solution does solve the problem is to run a test (usually an A/B test). Describe how you would structure the test including:
what’s a lightweight approach (vs a full blown solution),
what’s your target user segment to test with,
what metrics would you track (including success metrics and counter metrics)
what other teams/stakeholders/other parts of the ecosystem might be impacted and how you would engage them in the process
Decide: once you have the test outcome, how do you decide on next step. And this is sometimes leading right into the next part of the interview - trade off evaluation.
Answering Trade Off Questions
This type is trickier for me to put in step-by-step. Because as mentioned above, there are several types:
Trade off between feature A vs feature B
Trade off between metric A vs metric B (this could follow the feature evaluation question)
Trade off between config A vs config B of the same feature
etc.
But here are some general tips across these types:
Clearly understanding each option: including how the option works, and its impact on metrics and top level goals
Formulate a structured comparison between options: inform by #1, come up with a decision making framework with key metrics and dimensions. Make sure you focus on which has the biggest positive impact on to the top, and mention how you would go about gaining additional data to finalize the decision (e.g. running additional A/B test, how you interpret outcome).
It’s not binary: please know that sometimes it’s not necessarily always about picking option A vs option B. There could be option C (which could be a better version of option A or B, or even a dynamic decision on a per user basis).
Answering Troubleshooting Questions
When it comes to troubleshooting, I’d usually like to use the doctor analogy. In order for a doctor to determine the best treatment/course of action for the patient, she’d approach in the following steps:
Clearly understand the symptom
Pinpoint exactly where the symptom happens
Go through categories of possible root causes based on the identified symptom and location
And sometimes after understanding the symptoms, the doctor might determine that there’s nothing to worry about for now and will keep monitoring (and what should be monitored). If it’s indeed something to worry about, the specific root cause in the end will guide specific treatment and course of actions.
How does it apply to product troubleshooting question exactly?
Understand the symptom: understand the metrics precisely. Which exact metric(s) dropped? What’s the definition of that metric? by how much? Since when did it drop and what’s the (weekly/daily) pattern? What are some other metrics that might be impacted? How would you break down the given metric to further look into what exactly happened?
Pinpoint Symptom Surface: which user segment is impacted? Is it global or regional? Is it happening on a specific platform or across the board? Does it impact new or existing users or both? Which part of the funnel do we see the biggest drop? etc. etc.
Identify Root Cause: Based on the specific metrics and user segments that are impacted as identified above, start brainstorming possible root causes in different categories, that could cause the symptom. A good approach is to start from internal vs external factors. Internal includes recent product changes, outages, etc. External includes events, competition activities etc.
And of course, a good doctor does not launch into monologue, memorize rigid categories of root causes, and come up with dogmatic conclusion without carefully discussing with the patient along the way and asking the right questions. So be a good doctor by discussing with the interviewers along the way and flexibly adjust your next steps.
Reminders
To wrap this section, here are some reminders to help you optimize your product execution interviews:
Focus on “data-driven”: just another repeat, for you to focus more on quantitative decision making than qualitative user understand or fluffy strategic discussions, when you answer the question. The reason why it’s worth reiterating, is because we can all be misled into approaching it as a product sense question (depending on how the questions were asked).
It’s a collaboration: just like a product sense interview, there has to be a healthy back and forth discussion between you and the interviewer. It’s also a requirement to get inputs from the interviewer to move through the steps. It’s more than just “what do you think, am I on the right track” feedback, but also getting data from the interviewer along the way (e.g. is it impacting global user or one specific country?)
Peel the onion: Go one layer deeper at a time. Follow the pyramid principles. One of the biggest pitfall I see is to get to the bottle level details way too quickly.
Closing
That’s about it for now. It’s a lot in one article, and obviously there are nuances missing. I hope the above serves as a starter guide, but as always, don’t hesitate to reach out (johnny@introvertinproduct.com) for questions or if you need coaching assistance!
Next in the Interview Prep Deep Dive series I’ll look to cover other interview types, one at a time: Leadership/behavioral, Strategy, Estimation, Technical, and more. Subscribe if you haven’t to not miss a thing!