More businesses need to be like Chick-fil-A and In-n-Out. The menu is simple, the service is excellent, and the quality is high. Customers don’t have to learn a new menu each month, smile at apathetic employees, or brood over an incorrectly filled order. For those reasons, customers love them.
The lesson is clear: do less, do it well, and be consistent.
How else do you explain their durability? Both In-n-Out and Chick-fil-A were started in the 1940s, but growth hasn’t plateaued nor has their cult-like following. Chick-fil-A’s revenue tripled between 2010 and 2019. Meanwhile, the rest of the fast-food industry has attempted to compete in the market by serving up one ludicrous menu item after another. Carl’s Junior created fruit loop donuts. Taco Bell sold us Kit Kat quesadillas. It’s like the fast-food industry conducts strategic planning using junk food mad libs. And McDonald’s? They tried to make us eat salad!
As a product leader, you should strive to be like In-n-Out. If your customer cherishes you for beef, keep the beef coming. Don’t confuse them with lettuce topped with a paltry dose of carrot sticks and cucumbers. Resist the pressure to become something you’re not.
It’s unfortunate that most software companies, and the product leaders within them, don’t subscribe to the same ethos as these fast food icons. Successful software businesses begin as simple applications that succeed by delivering 10x greater value — relative to the alternatives on the market — with a limited set of features. Chrome won the browser wars because it is more secure, faster, and simpler than all other browsers on the market. Amazon became Amazon due to a multi-decade maniacal focus on selection, price, and convenience.
Being the best at a few things is a defensible strategy for building multi-billion dollar companies.
Myspace lost to Facebook because Facebook kept it simple. While Myspace layered on features, the product became slower, more difficult to use, and the primary value to the user — adding friends, sending them messages, and posting photos — was buried in an insidious medley known as “feature creep”.
Meanwhile, Facebook pulled off an opposing plan. New feature development was halted on several product lines and the bulk of the company turned its attention to a few initiatives — making the website fast and accessible from anywhere on the planet, ensuring users could easily find and add friends, and simplifying the user interface so that uploading photos and messaging others was trivial.
Within 18 months, Myspace went from the dominant social network on the planet to a dying business and Facebook went on to become a once in a generation company worth several hundred billion dollars. The flip flop was shocking and I was there to witness it.
Facebook’s focus on a few things that matter is what propelled it to the dominant social platform on the planet. However, their lack of focus in recent years has, ironically, turned it into a replica of a declining Myspace circa 2010. Users can donate to charity, find romantic partners, sell old furniture, play games, live stream video, send money, and find a job. But what happened to keep in touch with my closest friends and family? That was lost along the way.
Your challenge as a product leader is to create more value for the customer without introducing the clutter that obfuscates what you stand for.
What three things does your customer love you for? Is it speed, simplicity, and security? Convenience, low cost, and selection? Assuming you have this understanding, what plans do you have to enrich those values? And, most importantly, are you prepared to resist or kill the products that break your customer’s expectations of you? No, McDonald’s, I don’t want a salad.
Just because you can serve your customer a salad doesn’t mean you should. Apparently, Burger King has decided that the way to survive in the hamburger market is to serve something that isn’t even a real hamburger. What clearer sign do you need that a company has lost its way? By rolling out a meatless patty they are willing to concede the market to any competitor willing to stick to what most people want, which is a good-tasting meat patty.
Don’t be like Burger King. Don’t be like McDonald’s. Be like In-n-Out. Keep your product simple and its features aligned with the value your customers have come to know you for. Be fun. Be simple. Be consistent. Be convenient. Be low-cost. Be accessible. Be easy. Be a few of those things, but don’t try to be all of them. And no matter what, don’t be like Cheesecake Factory.
The majority of meetings are a waste of time. And in my opinion, one flavor of meeting that tops the charts in uselessness is the “status update” meeting. You know this meeting— the meeting where everyone gets together to share what they’ve been doing. It’s ironic that meetings like this exist because it gets in the way of people actually doing something productive. A cross-functional group of people (product, design, engineering, marketing and so on) working on a new product doesn’t greatly benefit from status updates.
I don’t claim to know the perfect recipe for a cross-functional product team meeting, yet I can say with a stiffened spine that the worst breed of status update meeting is the “stand up”, which has been popularized in agile methodology. I’m not the first to say this as others have had similar observations and provided comical opinions.
Most standups occur daily and some teams reduce the frequency to a few times a week. The majority of standups are short, yet often meander into casual conversation where people talk about things unrelated to work, like what they did over the weekend. None of this helps you build quality products at a fast pace. We’ve become so enamored with standups that popular tools like Slack even have a standup feature built into them.
But my stance on standups is the polar opposite of the trend. The only time in which a standup is truly beneficial is in the run up to a product launch. That’s when your troops are about to storm the beach and you want to make sure they are prepared. Are we ready to flip the switch on the A/B test? Is marketing ready to publish the blog post and push out the PR pieces? Does customer service have the scripts they need to answer customer questions that will come in right after launch? Do a standup when nearing launch to make sure the troops are ready to sack Normandy. That’s when it’s useful.
In any case prior to a launch, standups are mostly a waste of time. That’s because the bulk of the product development process centers around (1) building (2) making decisions. Point #1 is obvious (of course the team is busy creating stuff), but point #2 deserves a discussion, since that’s the meat and potatoes of the subject.
Hundreds of decisions need to be made during the development of a new product, like:
What’s making the cut for the MVP?
Who is the lead designer going to be?
How will we promote/merchandise this new feature?
Will we be A/B testing it?
What are our goals and key metrics?
Those are the obvious big decisions that nearly every product team needs to make. But there are also a long list of mid-sized and bite-sized decisions that need to be made along the way like:
Do we need to create a new UI component or can we use the one from our existing component library?
Is feature ‘X’ worth building since it balloons backend scope by 3 weeks?
Will this be a 50/50 A/B test or should we do a simple holdout group of 95/5?
Are we shipping on all three platforms in parallel across web, iOS, and Android?
Can we cut scope on Android since it’s only 5% of our users?
For anyone that’s built a product of reasonable scope/size, we all know that the set of decisions to make balloons as the project kicks off. It’s only then that you start to peel back the onion and realize the complexity of the task at hand, in addition to the set of decisions that must be made in order to unblock the project and keep it moving.
The universe of decisions to make begins quite large. As the product team progresses through those decisions, the hard (i.e. big) decisions tend to be made earlier in the project (e.g. “What’s making it into the MVP?”) and the mid-sized and smaller decisions tend to come in later in the game (e.g. “Can we simplify UI component ‘X’ to reduce the backend scope by a few days?”). The total number of decisions to make also approaches zero as a team gets closer to launch.
(Lots of big decisions to make early on with fewer and smaller decisions to make later)
But let’s recall the point I made a few seconds ago— the team is unblocked when decisions are made. “Unblock” is the magic word here.
Unanswered questions act as headwinds or speed bumps when building products. Using the classic example of MVP cutoff, a team must decide on which features will be included or excluded from an MVP. That’s a hard decision to make so product teams typically move slowly through this phase of development. This is one of the driving forces behind the creation of the Google Design Sprint methodology. As the methodology explains, you can use their design sprint method to “shortcut the endless debate cycle” to arrive at a few key decisions regarding what the first product prototype will be. Each unmade decision (i.e. each unanswered question) pumps the brakes on the development process as a whole. Protracted decision-making leads to protracted product development cycles, often by weeks or months (indicated by the red bars below). The Google Design Sprint method is one great example of how a team can make important decisions more quickly, which ushers them through the process of arriving at their first testable prototype.
(Answered questions lead to delays in development indicated by the red bars)
More on Standups
Before I share with you the meeting format that I’ve learned is most effective for product teams, I’ll harp a bit more on the ubiquitously popular “standup” meeting.
Let’s take a closer look at the prescribed format of a stand per agile methodology. Members of a standup (i.e. everyone on the team) are asked to share the following:
What did I work on yesterday?
What am I working on today?
What issues are blocking me?
There are several big issues with this format.
First, why should I care about what everyone worked on yesterday? Most of the time, you don’t care, and you shouldn’t care what others work on day-to-day. The practical reality is that you only care about what another team member worked on yesterday if it enables you to do your job (i.e. it unblocks you). Anything shared that does not explicitly unblock you is meaningless noise.
For example, a product manager may share something like, “Yesterday, I did one interview, worked on some specs, and interviewed one customer. Today, I need to interview one more design candidate and then I’ll update some of the user stories in the spec based on the questions I received from design. I’m not currently blocked on anything.” This example of a typical standup update is useless in terms of what it does to enable a product team to build a higher quality product at a faster pace.
Similarly, why should anyone care about what I’m working on today? Again, it only matters if what one person is doing today unblocks another person on the team. But if it’s really that important of a blocker, should that person wait until the following daily standup to make that known? Why not email that person or swing by their desk as soon as you’re running up against the roadblock and seek to remediate the issue as it arises?
Now, let’s dissect the third bullet point, which is to share what you’re blocked on. A team member may be blocked on something in one of two use cases. The first is that the task they are blocked on simply takes a lot of time. An example would be backend engineering waiting for more clarity on frontend specs before they can finalize the engineering design docs for what they must implement. The second reason why someone may be blocked is that a decision hasn’t yet been made. It’s less useful to say “I’m blocked by X” than it is to say “Hey, why don’t we get together and make a decision on X, so that we can move forward?”
Standups don’t carve out time for decision-making, which is the ultimate blocker. Rather, standups are designed to simply make the blocker known, yet not resolve the blocker. Resolution matters more than awareness.
A Simple Agenda to Keep Things Moving
I’d like to propose a simpler alternative that leads to more productive product meetings and improved product development cycles. It’s something I learned through trial and error in my career. I’ve found it to be the simplest, most effective meeting format for any product team (and for most meetings in general). Here it is:
Action items: who is handling what key action item and by what date?
What decisions need to be made?
I would run this meeting one or two times a week depending on the volume of decisions to be made. Sometimes as much as three times a week. Early in the development of a product, when a team needs to make many decisions, I increase the frequency of these meetings. As development progresses, the volume of decisions to make decreases and the proportion of time spent on building increases. Similarly, the magnitude of the decision (i.e. how hard or important it is) tends to decrease as development progresses.
If the team is effective at surfacing key decisions and in driving towards decisions expeditiously, then you can cut out weeks or months of unnecessary delays in the product development lifecycle. So, how do you make sure this is done?
Collecting Open Questions
To prepare for running effective product team meetings (i.e. decision-making meetings), I would go to each function lead on the product team (design, engineering, marketing, etc.) and ask them what decisions they needed made. I would collect the open questions and pull them into the agenda. It’s a simple enough task so you can ask people as you swing by their desk, ping them over email or Slack. In parallel, I would maintain a set of action items that came up during the decision-making meetings. From the set of open questions and action items I would compose an agenda and it would look like the following:
(Josh) Share latest iOS designs with Tammy by 9/13
(Deanna) Send marketing language for review to compliance by 9/15
(Andy) Sync with data science on how to configure the A/B test buckets by 9/15
Decisions to Make
Should we include the 3-5 days worth of UI design polish as part of the MVP launch or not?
Should we ask executive staff for support to get one more backend engineer or is there no parallel processing that can be done with extra resources to bring in the launch date?
Who is going to take the lead on starting customer development for the next milestone on our roadmap? Should we even start that yet or punt it by a week or two?
If I were leading the meeting, I would run through the set of action items to make sure we continue to execute on the critical tasks each of us signed up for and hold ourselves accountable to hitting our deadlines. Any new action items that came up during that portion of the discussion would then be added to the list in realtime. We would normally move through the action items portion of the agenda in less than 10 minutes.
The remainder of the meeting (which we normally reserved 60 minutes for) would then be spent on making decisions as a group. In many cases, a side conversation had already happened by a few relevant members of the team (e.g. design chatting with engineering about a particular aspect of the designs) and they could walk the team through the context of the open question and their recommended answer. In those cases, decisions were typically made in a matter of seconds or a few minutes. In minority cases (I would estimate about 30% of the time), the decision required ample discussion and might be too complex for a group setting and/or the time allotted.
In those cases, I would ask 2-3 of the most capable and relevant members of the team to form a quick working group either later that day or the following day to discuss the item and come up with a recommended decision. Their recommendation was then shared with the rest of the product team either over email, Slack, or in person at the next product team meeting. We effectively had multiple concurrent decision-making meetings going on in parallel, constantly driving towards reducing uncertainty and maintaining momentum.
(Use team meetings and small breakout discussions to quickly eliminate unanswered questions)
The more we had these meetings, the more effective we became at making decisions as a team, or in forming the breakout group to drive towards a decision and then close the loop with the rest of the team.
Something else to consider adopting is a decision log as part of this meeting format. I suggest using a single Google doc for maintaining a full record of all prior meeting agendas, as well as prior decisions made. That comes in handy when a post-mortem is run after the product has been launched to assess what the team did well or could have improved upon. Often, the full context of prior decisions made is lost, especially if those decisions are made in isolation by a few people and/or made several weeks or months in the past.
Maintaining a record of all prior decisions makes it very easy to reflect on the project during a blameless post mortem and the ability to identify the root cause of issues that eventually come up. Or, better yet, the log of all prior decisions made may help the team identify root cause behind a failed product launch. To make things convenient, I’ve created a copy of the agenda format and decision log that I used to use with my product teams. It’s publicly available for you to copy and use yourself. It’s simple enough but thought I’d share nonetheless.
The format for standups popularized in Agile methodology, unfortunately, isn’t well calibrated for driving efficient product development within a team. The root cause for long development cycles is that decisions weren’t made quickly and frequently. By replacing daily standups with less frequent decision-making meetings, product teams can save themselves lots of wasted time and build products much more quickly.
Over the course of my career, I’ve worked on product at Facebook, Twitter, Quora, and Wealthfront, and have advised dozens of other companies on the role of growth and product. Now, as an investor at Unusual Ventures, I get exposure to an even broader collection of companies, and the product cultures within them. Despite that exposure, there is a lot about the role of product that I still don’t understand. But what I can say for sure is that there are as many philosophies on the role of a product manager (PM) as there are letters in the alphabet. And we’re nowhere near getting to a point of agreement on the basic question of, “What makes for a great product manager?”
The confusion surrounding expectations has always stood out to me as one of the most unique and challenging aspects about this role. If you ask an engineer what they want from a product manager, they’ll give you an answer from their perspective. Some might say, “I want them to do all the stuff I don’t want to do, like take meeting notes.” Others might say, “I want them to be proficient with SQL, so that they can pull data.” Or, “I want somebody with good design sense.” Then, if you ask a designer what they want in a PM, you’ll probably get a different set of answers. Same thing when asking marketing or an executive of the company. If you ask the PM themselves (or Twitter for that matter) what they think a PM’s job is, you’ll get yet another set of perspectives.
These completely different expectations make the PM job very difficult relative to most other positions because what makes for a great PM can’t be narrowed down to one or two objective measures. PMs have a broader set of definitions and expectations to live up to. If you view each of these expectations as a set of circles, the world’s greatest PMs are those who are very good at a combination of these overlapping expectations. For example, you may have a PM who’s really great at writing SQL queries, which might earn them the respect of an engineer. But they also need to be excellent at conducting customer interviews and reflecting the voice of the customer to really impress designers. Ideally, you’d find somebody with an overlap of these skills. A baseline set of expectations for a junior to mid level product manager might look like this:
To make things even more complicated, these expectations can also vary from company to company. Some companies don’t even want to have product managers. That’s how conflicted the industry feels about the product management role. However, at the risk of being too prescriptive, I’ve laid out my framework for PM skills development below.
In my opinion, below are the most basic set of skills that a Product Manager should master in order to establish their role within a team:
Create Requirements — They can create clear product requirements that satisfy the preferences of the engineers and designers they work with (i.e. it unblocks design and engineering). There are many different approaches to product requirements, so there shouldn’t be a religion around a particular style. What matters most is that design and engineering feel enabled by the requirements a PM produces.
Customer Feedback — They can bring together basic customer insights through customer interviews, reading customer service tickets, etc. They do an adequate job of reflecting the baseline expectations of the customer they’re building the product for based on an explicit set of requests from the customer (e.g. the customer says a part of the product is broken).
Data Insights — They have some basic data proficiency where they can look at the data related to their product and potentially use that data to make decisions. They will have working knowledge of data languages like SQL and know how to use popular business intelligence tools like Looker, Google Analytics, and so on.
Run Meetings — They can run effective meetings with the team. That includes setting the agenda and tracking action items. For foundational abilities, this does not include driving decisions within the context of product meetings as driving decisions is more common amongst senior product leaders. I’ll touch on this more in a moment.
From what I’ve seen, having this baseline set of skills means you can be average-to-good and generally command some reasonable amount of respect from cross-functional peers in engineering and design. Then there are the additional skills that separate a beginner PM with foundational skills from a more experienced PM with intermediate capabilities.
To be an intermediate PM, you also need to layer on the following skills on top of the prior set:
Customer insights. Beyond collecting the plainly stated forms of customer feedback, this means deeply understanding the problems/needs of a customer and how that customer has historically solved those problems or met their needs. This PM can then diagnose how the product he/she is building will solve the customer’s unmet needs.
Basic design knowledge. Good PMs have working design knowledge so that they can distinguish between bad, mediocre, and good design. This predominantly comes down to UX fundamentals and not advanced topics like color theory. Does it require too many taps/clicks? Is the product copy unclear? Does it meet the stipulated customer needs? Is the cognitive load too high? Is the information hierarchy correct? These are the sorts of questions that a good product manager will ask when working through prototypes.
To be an exceptional PM, you must combine the previously mentioned skills with a few more skills. This person can operate at the executive level or be the product lead on the biggest, most important product initiatives within a company.
Customer Innovation. This PM not only knows the explicit customer requests and has intimate knowledge of the customer’s needs, but can introduce new products or features that exceeds the customer’s expectations and innovates on the customer’s behalf.
Vision and Strategy. Exceptional PMs can also internalize founder-level vision for the company and turn that vision into an actionable strategy. They have the most keen line of sight into the series of products/features that need to be built to achieve the founder’s vision. And once they have the vision and strategy clearly understood and explained to their team, they can enforce great execution against the vision and strategy.
Broker hard decisions. The best product managers are those that can also drive a team toward making a high quality decision in a world of uncertainty. Foundational and intermediate PMs can arrange and run the meeting, but aren’t necessarily great at driving a product team towards a decision in difficult situations. The best PMs are those that everyone else in the room looks to when making a hard decision and they typically make the best tradeoff decisions given the circumstances.
As you can see, the requirements for becoming an exceptional product manager are high and numerous. Going back to the original venn diagram analogy, exceptional product managers need to have a lot of overlapping skills and each function within the company places particular emphasis on some of those skills. Engineers might assess a PM based on data proficiency and quality of requirements, design might assess based on customer-centricity and some degree of design thinking, while executives may assess a PM based on the ability to lead a product team through complex initiatives and fidelity of vision and strategy. Across the board, everyone is expecting a PM to be a good decision maker.
A needle in a haystack?
That said, it’s no surprise to me that I often hear founders gripe about how they can’t find great product managers. They often say they only come across average PMs. As part of that, I arm them with the above framework so that they can be more specific when it comes to critiquing the caliber of their product managers, in particular once they’ve adjusted their expectations for the level of seniority.
When I dig in more deeply, what they are really saying is that it’s very hard for them to find product managers with the exceptional PM qualities of customer innovation, clarity of vision and strategy, and ability to make hard decisions. Unfortunately for PMs, most founders expect them to have the skillset of an exceptional PM, regardless of their level of experience. This triumvirate of skills (customer innovation, vision and strategy, and ability to make hard decisions) is what I call “Founder’s Feel”.
For example, when Facebook was working on Messenger, it first started out as an integrated messaging app as part of the web client. One day, Mark Zuckerberg declared that it must be made a standalone mobile app and that Facebook users would be required to download it if they wanted to chat directly with one another. That wasn’t an obvious decision at the time, certainly to most of the rank and file employees, yet it was the correct decision given the pressure Facebook was feeling amongst a growing collection of competing messaging apps. Facebook was being unbundled by competing products so they decided to get ahead of it and unbundle themselves. This is the sort of decision-making that great founders make and, sometimes, exceptional product managers can make as well.
I’ll give you another example from my time at Wealthfront. We had just launched our mobile app and the number one source of customer feedback we were getting was that customers wanted to see their daily returns front and center in the mobile app. In other words, customers wanted to be able to log in and see whether their account was up or down relative to the previous day, and they wanted that front and center in the app. I remember telling our product manager at the time that we were never going to do that. The PM looked at me like I was an alien and kept repeating that customers were asking for the feature.
I explained that customers were asking for that feature based on what they thought Wealthfront was at that point in time, which was a tool to manage their investments. But what we were going to be a few years into the future would be their holistic money manager where we would handle their financial planning, banking, investing and so on. Holistic money management is not about showing daily returns because whether the market goes up or down 1% per day does not matter in the long run. Most customers were investing and planning for something several years or even decades down the road. I pointed out earlier that a great product manager understands customer feedback first and foremost. But an exceptional PM knows how to contextualize that customer need within the broader company vision and strategy, and then decide whether or not to implement what the customer is asking for.
Fast forward a couple years, we took daily returns and relegated it to a lower position within the app and put long-term financial planning front and center for the customer. Eventually, we saw that the savings of customers who engaged in that long-term financial planning feature went up by a double digit percentage. Sometimes, listening to explicit requests from the customer can lead you in the wrong direction, as would have been the case here.
Developing “Founder’s Feel”
So what does it take to develop “Founder’s Feel” in order to become an exceptional product manager? How does one go about building strong instincts around customer innovation, a vision for the future, and a knack for making high quality decisions that others don’t necessarily see? The bedrock of it is in knowing the customer better than anyone else.
The way I think of it is that you’re collecting unstructured data every time you have a conversation with a customer. It’s not structured like the data you retrieve from a database. Rather, you’re looking for structure (i.e. patterns) in the data that only becomes apparent once you’ve had enough customer conversations. For example, imagine that an insight from a customer conversation is a data point that you plotted on some nebulous graph, like I’ve shown below. Each time you have a conversation you collect insights and start plotting them into groups based on similarities/themes. Once you’ve had enough conversations, the themes start to cluster around common insights or customer needs.
Once you hone in on the clusters they start to make sense to you and “feel” obvious, especially after you’ve had enough conversations with a customer. In the below example, each cluster outlined in red could lead to a meaningful product decision.
Each time you talk to a customer, you capture unstructured data. If you have enough conversations with customers, that unstructured data starts to take shape in the form of extremely strong instincts around potential for customer innovation and product vision that seems less intuitive to others. That’s what “Founder’s Feel” is.
The founders of Wealthfront talked to hundreds of initial customers face-to-face and read most customer service tickets from the first couple thousand customers. Several years after founding the company, the founders continue to read customer service tickets and talk to customers face-to-face. When you compare their ability to make a decision in an uncertain environment vs. a product manager who hasn’t talked to any customers, there’s no comparison to be made. They have a “feel” for what to build (and not build) for Wealthfront customers that is unmatched due in large part to the sheer volume of unstructured information they have collected over many years while building the company.
Returning to the original question: What makes for a good PM? The answer is it’s really complicated.
To start with, good PMs can navigate ambiguous expectations and can apply their skills based on the team’s needs. They know the different levers to pull depending on who they’re working with and can adapt accordingly. One thing that can’t be sacrificed is knowing what the customer needs better than anyone else. This means doing customer research, design testing, and diving into customer support tickets, spending time sitting with the customer support team, and so on.
If you’re just getting started as a PM, focus on building the foundational skills I outlined above. That’ll be the foundation through which your cross-functional peers see the value you add. With time and practice, you can hone the additional skill set I’ve outlined that can make you an intermediate or exceptional PM. But if you’re not good at the fundamentals to begin with, you’ll lose the respect and trust of your team and they might wonder, “What is this person’s job exactly?”
For senior PMs, you have to have the ability to broker decisions in situations where it’s hard to make the right call and drive alignment around the vision and strategy. In addition to that, you can weigh in on product decisions using some working design knowledge. I’m not talking about weighing in on square corners versus rounded corners. I’m talking about the overall flow of products, if the experience is clear or cumbersome, and if it meets or exceeds customer’s needs or expectations.
Finally, the rarest of PMs develop their own “Founder’s Feel” — i.e. a knack for making the best strategic decision that’s informed by a top rate understanding of the business, customer, and the market. They know how to work under uncertain conditions and get a sense of the bigger picture. Those exceptional PMs can translate the founder or CEO’s vision into a clear execution plan.
With all this in mind, though,my default position is that the vast majority of product teams are better off being led by a small group of cross-functional owners as opposed to the outdated notion of “the PM as the CEO of the product”. The most effective product teams are those where the leads on Product, Design, and Engineering (and sometimes other functions like marketing, customer service, and data science) are all mutually bought in to being experts at understanding the vision, the customer, and have shared ownership of the outcome. This means that a Product Manager should often defer to Engineering or Design when key decisions need to be made, specifically when those other functions are better fit to make those decisions. And that’s often the case! I would argue that a great PM also shows “Founder’s Feel” by referring to the right person in the room to make the decision.
In my experience, small group decision making typically leads to vastly better outcomes and if you have a true product leader, that person emerges naturally just like any leader does. There’s no explicit conversation about “who is in charge” of the group. Rather, they do such a good job that the peers around them see it, they know it, and there’s not a discussion about it. That person simply rises to the occasion. It’s the person that everyone in the room naturally turns to for the hard decisions and the mark of an exceptional PM.
“Our industry does not respect tradition— it only respects innovation.”
That’s what Satya Nadella wrote in his opening email to the company shortly after becoming Microsoft’s new CEO. It was a clear call to arms that Microsoft needed to reignite innovation in order to scale the company after roughly 15 years of stagnation. The price of Microsoft’s stock has increased ~3x since he came back because the market seems pleased with Microsoft’s sharpened focus, progress made in the cloud business, and willingness to change how it used to do things in order to compete in the future. Some of this could be window dressing or marketing speak, but the changes happening at Microsoft seem genuine.
Satya said nothing about doubling down on what’s already working in order to get more juice out of the squeeze. Rather, he ended the email by emphasizing the need for clarity of focus on new innovations and on changing the culture which, for the most part, was focused on preserving the status quo for over a decade. It’s not unheard of that a large company often forgets how to innovate.
I haven’t spent enough time at companies with 1,000+ employees to speak deeply about the dynamics of large company stagnation, but I can speak to it happening at early-stage startups. In particular, I find it interesting that the same two problems Satya outlined for Microsoft often appear within early stage startups as well: i.e. the culture becomes comfortable with the status quo and the company loses its ability to innovate.
How does it happen? When a startup becomes obsessed with and designed around data and optimization. Today, every 50 – 100+ person startup has multiple business intelligence tools, off-the-shelf A/B testing tools, a data science team, and product managers who know much more about writing SQL than they do about interviewing customers.
In fact, I kept score while interviewing PM candidates in 2017. I spoke with 67 product managers. About 50 of them were reasonably proficient in SQL and could write a few queries on the spot. Guess how many knew how to conduct customer development? Three. That’s it. Only three product managers could proficiently describe the purpose, process, and outcomes from customer development. 75% could write SQL, but only 4% knew how to properly interview a customer. It’s a small sample size, but the gap is large.
Here’s why that’s bad: Most startups, just like large companies, need to go through continuous phases of innovation in order to create 2x+ step changes in the potential for their business. The process of going from 0 to 1 with their first product is an innovation. It’s what allows the company to get off the ground. Sometimes, that original innovation is enough to carry them from seed to IPO. But that is incredibly rare. What’s more common is that startups need to innovate several times over in order to create step changes that help them scale from early stage to growth stage and from growth stage to a publicly traded company.
Over the last 10 years, there has been a massive overcorrection in the direction of optimization based on broad availability of data, leading me to find that most PMs are incapable of effectively deriving insights from customer conversations and most startups are incapable of producing new product innovations beyond the initial product that they take to market. They’re great at A/B testing, but not great at creating new features based on customer insights and a leap of faith.
To put it plainly, growing through data analysis and A/B testing isn’t the only path to future growth. While it seems obvious, I see very few startups designed for innovation, which may be the biggest driver to new growth for your business. Do you think Facebook would be at its current scale without innovations like News Feed? Community-driven translations to expand globally? Or the developers platform? The answer is obviously “no”. Take a look at MAU acceleration beginning in 2007 / 2008. That coincides with the launch of the international translations app, which allow Facebook users to crowdsource the translation of the product. It took several months to build and a few years of ongoing maintenance and development to mature the product. That innovation led to a boom in active user growth.
The point I’m making is that today’s startups very quickly fall into the optimization trap where they think future growth will largely come from optimizing their existing product. The better approach is finding the right balance between optimization and innovation since both methods can produce future growth.
By the time you’re done with this series of blog posts, you’ll have the knowledge and tools you need to do the following:
Design a company-wide org chart that creates an explicit balance between optimization efforts and innovation efforts
Wisely select the “right” types of experiments to run to increase your chances of improving growth through optimization
Implement a repeatable product development process for creating new, innovative features
Optimization Versus Innovation
We should first start with a more detailed explanation of the difference between optimization and innovation. Optimization is when a startup iterates on its existing products or services to squeeze more juice out of the orange. Typically, the results of optimization are incremental in nature.
If they are incremental in nature, then why do them? Well, because many small optimizations can accrue into large long-term results when you allow those optimizations to compound.
Here’s a simple example. In the below graph, I compare the 12 month growth in monthly active users (MAUs) in 4 hypothetical cases. The blue line is the base case where the monthly growth rate is slowly declining, leading to flattening growth. The red line is for sustained 10% month-over-month growth (MoM), yellow is sustained 12% MoM, and green is sustained 14% MoM. If a startup can optimize its way towards a slightly higher and sustained rate of growth, the compounded outcome is very different relative to the base case. In fact, this is what we did in 2009 at Facebook. Our growth team focused on optimizing our way towards a sustained 2% week-over-week growth rate because we knew that we would grow from ~100 million MAUs to ~300 million MAUs in 12 months if we did so. This happened to be the company-wide goal for that year.
Innovation is when a company embarks on building entirely new products or services for existing customers or for a new segment of customers. Innovation can also involve expanding into an entirely new business line. However, this happens so rarely (hello, Amazon!) that I won’t focus on this definition for the time being. Additionally, innovation can create step change improvements in the trajectory of the company, although they are much more difficult to discover and successfully execute on.
I’ve taken the same scenario above, but added in a 5th option which is labeled as “with innovation” in the below graph. What this does is take the base growth rate scenario and applies a 2x multiplier to growth midway through the year (e.g. you build a new feature, such as Facebook’s News Feed and it leads to a step change in monthly active usage). This assumes no optimizations along the way.
The point isn’t that you should pick one approach to growth over the other. Rather, the ideal outcome (and most realistic) is a healthy combination of both optimization and innovation. In the below scenario, I assumed that a segment of the company is working on optimizing the existing products and services to sustain 10% MoM growth and another segment is working on new product innovation that leads to a 50% bump in MAUs midway through the year. This scenario is plotted as a black dashed line on the graph.
Picking a Path
The appropriate question to ask is, “For my company, should I be innovating or optimizing?”
For Seed and Series A startups the practical reality is that you are headcount constrained into picking one over the other because you’ll have less than 20 employees. Prior to establishing product market fit, you’ll be entirely focused on innovation because you’ve yet to figure out the new technology that delivers something better, faster, cheaper, and more convenient relative to the alternatives in the market. Consequently, you’ll have very little growth or customers to optimize on top of, so don’t waste your time optimizing if you don’t already have exponential organic growth.
As a company matures to the point of Series B and beyond (sometimes with a large Series A) it can hire enough people that it can contemplate doing more than one thing at a time. From my experience that’s at the point in which a consumer software company has 30 or more employees. On average, about half of the employees will be engineers, so that means you’ll have 15 people that can do the building. With 15 people doing the building you can divide them amongst 3-4 teams— e.g. 2 product teams, an infrastructure team, and a floating pool of engineers needed for miscellaneous tasks and on-call work.
When a company reaches 100 employees it can certainly multi-task. Its 50 engineers can be subdivided amongst 2-3 well-staffed product teams, 2-3 infrastructure teams, and still be able to manage on-call support and miscellaneous tasks.
Stocks and Bonds
Assuming a company is able to reach the scale of 30+ employees and is now capable of walking and chewing gum at the same time, the question becomes, “How do you allocate those people in terms of optimization versus innovation?” I like to use investing analogies when thinking through this decision.
Most investors should have an investment portfolio that maximizes their returns given the amount of risk that is appropriate for them to take (this concept is known as Modern Portfolio Theory). Put in simple terms, it stipulates that you’ll want a diversified portfolio comprised of a mix of higher risk, higher return investments (e.g. stocks) and lower risk, lower return investments (e.g. bonds). Depending on the level of risk you can afford to take, you’ll want to shift the allocation towards certain investments and away from others. For example, if I’m 70 and ready to retire, I should be taking very little risk and will want a portfolio weighted heavily towards low risk, low return investments (bonds). If I’m 30 and putting money into a retirement account that I’ll use 30 to 40 years from now, then I should be taking on more risk to generate more returns during that long time horizon (i.e. more stocks).
I hope you are starting to see how this investing analogy applies to your startup thinking. Innovation is your stocks and optimization is your bonds. The question to ask is, “What proportion of my company’s focus should be on optimization versus innovation?”
If you’re building a seed stage startup, then you’ll solely be focused on innovation (all stocks and no bonds) because you’re trying to build something new and innovative that finds product market fit. If you’re working on a series A or series B startup with clear indicators of product market fit (i.e. exponential organic growth), then you should be considering the trade-off between optimization and innovation.
Facebook is a good example of optimization and innovation at play. While I was at the company (2008-2010), we did a bit of both. The Growth Team was focused predominantly on optimization by improving sign up conversion rates, new user onboarding, reactivated user onboarding, getting people to add more friends, and a vast library of miscellaneous A/B tests for the sake of getting more users. Meanwhile, several of the core product teams were pushing out big innovations like the first smartphone app, various News Feed innovations, large enhancements to photos, and the developer’s platform.
In the next part in this series I’ll discuss how you can design an org chart and product teams that create an explicit balance between optimization and innovation. If you’re ready for that, go ahead and jump right in. And for broader context, here’s a list of all four parts in this series.
There’s a powerful concept known as “shipping the org chart”. It was brilliantly outlined by Steven Sinofsky in his piece on Functional vs Unit Organizations. The TL;DR is that the design of your org makes its way into your product. In other words, your product is significantly influenced by the nature of the organization you’ve designed within your company.
Here’s an example from an org chart I recently reviewed with a Series A (soon to be Series B) startup currently scaling from 15 employees to about 45.
It’s a fairly straightforward org design. The ops team is focused on optimizing the field operations folks to scale their service at lower cost. The eng team is building out and scaling underlying services and products to support 10x growth in a number of customers. There are two product team. The first team is the LTV team, which is focused on increasing revenue per user. The second team— the growth team— is focused on improving all important conversion rates, such as sign up rate, new user onboarding, and so on. Lastly, the marketing team is focused on acquiring more customers.
That all seems reasonable— but, with one catch.
I asked the founders of this Series A company, “Who is focused on delivering more value to the customer?” To which I received a blank stare, followed by a bit of head scratching, and then a final, “Uhhhh … well…good question!”
The problem with an org chart like the one above is that it’s almost exclusively aligned with producing value for the business— so much so that very little attention is being given to satisfying the needs of the customer. Here’s where things get really tricky—it also pushes the company deep into optimization territory. To be specific, it’s the design of the product teams (those highlighted in green) that is most worrisome. I’ll elaborate more on this in the next section.
Imagine you have 1 junior/mid experience product manager, 1 junior/mid experience designer, and 2-3 engineers— each with a few years of experience. That’s a fairly common atomic unit of a product team within a startup. This small team now refers to themselves as the “LTV team” with an understanding that their primary metric is to improve revenue per customer. The next step for them is creating a roadmap, which they begin to do through the lens of increasing revenue per user to maximize LTV for the business.
The very first project that the team puts on their roadmap is to A/B test the pricing tiers for their subscription business. Another item on their roadmap is to A/B test variations of the subscription cancellation flow with alternative messaging and discount offers in an attempt to convince customers to not cancel their subscription. Following that, the team has fleshed out a portion of their roadmap for testing new email, in-product, and push notifications to encourage freemium users to upgrade to one of the paid tiers. Again, these are all reasonable projects to work on. The issue is that they are all focused on incremental optimizations for the benefit of the business and don’t add any additional value to the user. This is the slippery slope I alluded to a few paragraphs ago.
Fast forward 12 months and the LTV team is still busy running A/B tests, looking at funnel data, and squeezing out 5% – 10% wins via the occasionally successful experiment. Meanwhile, they haven’t shipped any new, innovative products or features that deliver substantial value to the customer (which can also increase LTV for the customer!). While exercising their data analysis and A/B testing muscles, their customer development and new product development skills have atrophied.
Jump ahead another 6-12 months and this team of highly skilled optimizers is scratching their head because the company is lagging its growth goals. They’ve continued to hire PMs whose strength is in running SQL queries and designing experiments. They’re finding the occasional 5% – 10% win, but they’re starting to get the sense that they’ve scraped the bottom of the barrel because it’s becoming increasingly hard to find a positive experiment. Meanwhile, one of their competitors is scaling more quickly, compelling them to want to run even more experiments because they’re questioning if they just haven’t run the right A/B tests yet. Anecdotally, many employees at the company notice that the amount of customer love they received on social media has slowed down. They observe a noticeable decline in feature requests and praise from their existing customers in Zendesk as well.
Meanwhile, the Growth team has been busy doing much of the same. They’ve been running experiments, building innumerable data dashboards, and commiserating with the startup’s lone data scientist as to why growth is below plan and becoming increasingly dire, despite having run dozens or hundreds of A/B tests over the last two years. Several of the tests were successful, but what gives? Why does growth suck relative to their expectations?
The product teams and company have entered what I like to call “optimizers purgatory.” They’re in a strange middle ground between succeeding with plenty of data and A/B testing abilities, minus a single meaningful innovation to the user experience in the last year or two. This sounds like an extreme hypothetical, but it’s incredibly common. I’ve personally been there and worked with dozens of other startups that have encountered optimizers purgatory as well.
What can be done? The company could have considered an alternative to the org chart that struck a better balance between having some focus on optimizing for business value and some on innovation for customer satisfaction. This may in turn create business value far greater than the value that comes from solely optimizing for business metrics. Below is an example alternative that swaps the LTV Team for a Client Value Team. This new team’s primary metric is customer satisfaction score— e.g. the percent of customers “very satisfied” with their experience.
Take the same atomic unit of a team (1 PM, 1 designers, a few engineers) and you’ll find their roadmap is wildly different than the LTV Team’s roadmap. This difference is simply because their team name implies creating new value for the customer and their primary metric requires that they increase customer satisfaction. Recall the LTV Team had a roadmap full of A/B tests focused on optimizing the business metrics. The Client Value Team’s roadmap is more likely to contain a list of new, high value features that customers have been asking for and new, innovative value that customers weren’t expecting to receive, but will be delighted with.
In contrast to the LTV Team, the Client Value Team will develop their customer development and product development muscles. They’ll have well-defined customer research and design research methods. They’ll likely also develop a closer relationship with the customer service employees within the company, leading to regular meetings with the head of customer service where they review the latest Zendesk customer requests. They’ll have fewer data dashboards and won’t be able to speak as eloquently about the parts of the product that are well optimized, but they will be able to speak about which customer complaints have tapered off and which new customer requests have bubbled to the surface.
The LTV Team and the Customer Value Team have become two very distinct organisms, simply because of the name of the team and the type of metric chosen— i.e. a customer success metric versus a business success metric. This is the notion of “shipping the org chart” at play and it’s an essential concept to understand when thinking about designing an organization with the intent to grow the business.
Creating a Balanced Organization
When working with founders on creating an org chart that adequately balances growth from optimization and innovation, I give them the following exercise:
Step 1: Concisely describe your mission and vision for the next 2-3 years
Step 2: List the 2-3 things that must be true for your customer to realize that vision
Step 3: Design an org with product teams that map to the 2-3 truths for your customer
Step 4: Revise and edit until satisfied with the results
Here’s a practical example from Wealthfront, where I was most recently the President:
Step 1: Wealthfront’s mission is to provide everyone access to sophisticated financial services with the vision that our customers would use Wealthfront to exclusively manage all of their finances.
Step 2: In order for that mission and vision to be true, our clients would need to (1) create a free financial plan that captures their needs and wants; (2) have a superior set of banking products relative to what they could get at large banks; (3) have world-class investment management that’s typically only available to the ultra wealthy.
Step 3: We set out to design the primitives of a product organization that reflected 1 and 2 above. It looked something like this:
We came up with an Onboarding Team that would digitize many of the financial processes traditionally handled over the phone or via paperwork. By digitizing these experiences, we could ensure “everyone gets access”, per our mission statement. The Onboarding Team’s primary metric was customer satisfaction. For this metric, they measured the percent of users that were very satisfied with various parts of the onboarding experience. We made the leap of faith that if the customer was more satisfied with the experience, they would trust us with more of their money ( our data science team proved to be true). That ensured we took a very customer-centric approach to innovating with the onboarding experience.
Secondly, we created a Financial Planning team to build out a whole new suite of products, so that our clients could get more value out of Wealthfront beyond just investment management (the company began with this offering). Finally, we had a Financial Services Team that would build the next generation of investing and banking products, so that our clients could get access to financial products typically reserved for the rich.
Step 4: Once we had those teams in place with a clear charter for creating new innovative products (as opposed to simply optimizing the products we already provided), we put the rest of the company org in place.
And within the product organizations, we could then provide guidance on the proportion of their roadmaps/time and effort spent on creating new feature innovations versus optimizing for growth with the existing feature set. For example, one might ask each product team to construct roadmaps that are 70% focused on building new value to the customer and 30% spent testing and optimizing for key business metrics related to their product line. With this approach to org design, a startup can be very explicit with its allocation towards growth through both optimization and innovation.
Another version of striking a balance between optimization and innovation is as follows: In this case there are 3 innovation-focused product teams (in blue) and 1 product team (the growth team in green) that is focused exclusively on optimizing the existing features and experiences in order to improve the business metrics. This would lend itself to a split of 75% innovation and 25% optimization.
As noted earlier, companies need to pick their balance of “stocks and bonds”— i.e. their mix of optimization and innovation. However, they shouldn’t pick their mix once and set it for perpetuity. The mix should change over time depending on the circumstances of the business.
For example, if your company launched a new product line a few months ago and is experiencing exponential organic adoption, then the product clearly has product-market fit within your customer base. It may make sense for that product team to then spend 3-6 months optimizing the existing features within that product line to maximize for adoption via some low hanging fruit experiments. This is especially true for network effects businesses since optimizing the drivers of the network effects can produce massive results. That was the case at Facebook where we spent a lot of time optimizing for sign up rate, new user onboarding, and getting people to add friends. By doing so we meaningfully accelerated the growth of the company due to it being a network effects business.
Conversely — and is the more common scenario I’ve seen at early stage startups — is that topline growth has stagnated as a result of having not shipped anything new and innovative in the last 1-2 years. That’s often the case since most businesses do not have a network effect and must therefore grow through new product innovation. The following example comes from my time at Wealthfront. At one point three out of four product teams were setup to focus mostly on new product innovation (Onboarding, Financial Planning, Financial Services) and one team was set up exclusively for optimization (Growth). Within the Onboarding, Financial Planning, and Financial Services roadmaps, the teams then have an explicit balance of how much of their efforts is dedicated to building new innovative features versus optimizing the existing products.
In subsequent quarters, the mix would change based on new insights or overall changes to the business. The key point is to remain flexible and use this simple mental model of “stocks and bonds” to regularly communicate and decide the appropriate mix of optimization and innovation across the company and within each product team’s roadmap.
If you want to take a stab at designing your own org chart using a similar process, go ahead and copy this free template that I made available and create a version of your own. It provides guidelines for laying out your org chart, listing what you must accomplish for your customers in order to realize your mission and vision. It’s also a place for you to balance optimization and innovation within each roadmap, as well as list the customer success metrics for each innovation team.
In the next part in this series I’ll discuss product development for optimization. The most important aspect of it is choosing the “right” experiments to run. I’ll do a deep dive on what that means and provide guidelines for you to use when it comes to making your own experiment selections. If you’re ready for it, go ahead and give it a read right now. And here’s a list of all four parts in this series in case you want to jump around.
Assuming you’ve determined the right balance of optimization and innovation from the above sections, we can now take a closer look at how to manage an optimization roadmap and pick the “right” experiments to run.
Creating a Roadmap
Like any good product team, you should begin with a roadmap. The roadmap should be organized in priority order with the priority determined by estimated impact and level of effort. For example, if you estimate that a certain set of tests can produce a large increase (double digit gain) in the metrics for a relatively small amount of effort (a few weeks or less of engineering and design support), then it’s likely a high priority experiment. I’ve also created a template for creating your own experimentation roadmap, which you’re welcome to make a copy of and run with it.
The roadmap has two segments to it: The first segment allows for estimating the impact of various experiments so that you can rank them in priority order. The second segment is intended to capture the results from the experiment. It’s essential to maintain a history of all experiment results so the team can conduct post mortems in order to refine their experiment selection and design.
Generally speaking, I recommend that optimization teams— such as a growth team—operate in 6-8 week sprints focused on improving one metric at a time. A common mistake I see is a small growth team trying to optimize multiple metrics in parallel. This lack of focus normally leads to subpar results. In contrast, significant results can be produced when the full weight of a growth team is poured into a single metric for at least a few months. The team will find that they improve their pattern recognition through focused effort, leading to better test results as time goes on. As an example, during my time at Quora, our growth team spent 16 months optimizing solely for sign up rate. During that time frame we increased the sign up rate from SEO traffic from 0.1% to north of 4%. Once we reached the bottom of the barrel on that particular metric, we moved onto the next metric and repeated the process. To encourage this type of focus, I broke the experimentation roadmap template into multiple tabs where each tab maps to a roadmap for a specific growth metric — e.g. churn vs. reactivation vs. signups and so on.
Picking the “Right” Experiment
Picking the right experiment to run is part art and science. By art I mean using judgement to craft a user experience worth testing. By science I’m referring to the practical constraints of testing new experiments on a relatively small population (i.e. sample size in statistics speak) when you’re still an early stage startup.
I often see startups try to run A/B tests in the same way that large companies like Google and Facebook do. They create a list of A/B test ideas that require fairly limited level of effort and then they start shipping dozens of small change tests fairly quickly. A classic example would be making changes to the call-to-action on a landing page, such as on the homepage, and perhaps testing the location of the call-to-action as well. The problem with this sort of test is that a startup often has a much smaller sample size (because they have less traffic or users of the product), so running and resolving that A/B test at high statistical confidence takes much, much longer than running a similar test at a high traffic product like Facebook. The relationship between experiment thoughtfulness and sample size is captured in the below diagram.
Here’s how to interpret it: Companies with a large sample size (a lot of traffic) don’t have to be as thoughtful with experiment selection and design. The reason is that the large company can make relatively small changes to the product, set up an A/B test to measure the effect, and then resolve the experiment in a matter of days at high statistical confidence because they have a wealth of data to lean on. On the other hand, a small startup with very little traffic (small sample size) needs to be much more thoughtful about experiment selection and design because an A/B test on a small sample size that produces a small change relative to the control will take weeks or months to harvest enough data to reach a statistically significant conclusion. I’ll demonstrate this effect in the below table.
Let’s imagine we have three different startups (A, B, and C — below). Each is going to run an A/B test on their homepage where the base conversion rate is 10%, the relative increase in conversion rate they are aiming for is 5%, leading to a new conversion rate of 10.5%. However, each startup has a different volume of daily traffic. Startup A receives 100 visits per day to the homepage, B receives 1,000 visits per day, and C receives 10,000 visits per day. Using the A/B testing calculator from AB Tasty to calculate the necessary test duration, we get the following results.
You can see from the data that the test duration declines significantly as a result of having more samples (i.e. traffic) in the test funnel. Now, let’s take a look at what happens when you tweak the magnitude of the relative experiment effect. In other words, when you run a test that produces a small, medium, or large change to the baseline conversion rate.
By increasing the magnitude of the relative experiment effect, the test duration declines precipitously. The key takeaway here is to aim for large changes. That seems like an obvious observation, yet I see many startups testing relatively minor changes to their product in the hopes it will produce a double digit increase in the target metric.
Finally, let’s look at what happens if we manipulate the base conversion rate. By base conversion rate I’m referring to the starting conversion rate. For example, if you have 100 visitors/day to your homepage and 1 user signs up, and you’re running an A/B test on the homepage, then you have a base conversion rate of 1%. If instead you run an A/B test midway through the sign up flow where there are 10 visitors per day, and 1 visitor manages to sign up at the end of the flow, then you have a 10% base conversion rate. What you’ll notice in the below scenario is that test duration decreases as a result of having a higher base conversion rate. Practically speaking, that means you’re more likely to reach statistical significance quicker if you A/B test in the bottom half of a funnel versus the top half since the bottom half has a higher base conversion rate.
To recap, there are a few key lessons to take away from the above scenarios:
Smaller startups can’t test like big companies because of sample size limitations. They simply don’t have as much traffic. If they try to test small changes to the product, which produces a small relative change in conversion rate on an already small sample size, then the test will take months or years to conclude. Startups don’t have the luxury of waiting around for insignificant results like that. On the contrary, startups need to produce step change increases in their rate of growth in order to achieve liftoff and set themselves up for another funding round.
Startups must test big changes to their product in order to manage sample size limitations. If a startup runs an A/B test for a significant product change that leads to a 30% worse conversion rate, they’ll find out in a matter of days and can quickly kill the experiment and limit the downside. If it turns out that the test produces a 30% increase in conversion rate, the company will also find out in a matter of days and can turn it live to 100% of users and experience a large increase in its rate of growth. When you think of it that way, the startup really has nothing to lose!
The bottom half of a funnel is often a better place to test than the top half of a funnel because obtaining statistical significance on a high baseline conversion rate is more likely than on a low baseline conversion rate.
It’s essential that anyone working on an experimentation team or roadmap understands the above statistical concepts. If so, they are less likely to stack their roadmap with poorly chosen A/B tests that will take too long to run and produce results too small to change the trajectory of the company.
In the last part in the series I’ll do a deep dive into how to implement a repeatable product development process that improves the chances that you can ship new innovative products at a fast pace. Below is the full list of posts in the series in case you’d like to hop around.
Modern software companies follow a variety of common conventions to scale quickly and efficiently. For example, most software companies have a defined and documented approach for engineers when it comes to writing, reviewing, editing, and deploying new code. It’s important to settle on some standards and procedures for software development because it means a company can write code quicker, reduce mistakes that are inherent in writing code, and provide a better working environment for software developers. The end result is more and better products delivered to the customer, which in turn is good for the business.
However, standardization of a product development process is uncommon within startups. Most companies lack a clear procedure for taking an idea and turning it into a high quality, shippable product. What typically happens is product teams form and are left on their own to figure out how they want to drive new product development. For example, who is responsible for conducting customer research, when, and how should it be conducted? How does a team come up with an initial prototype for a new product? How do you iterate on it over time? In what ways can you maintain clear internal communication with key stakeholders as the product is being built? When and how do you come up with the go-to-market plan for the product? A well-designed product development process will have an answer for each of these questions and will help you ship more and better products to your customers. Without such standards, each product team will build products through different methods, leading to inconsistent product delivery timelines and inconsistent product quality. The last thing a startup needs is more unpredictability.
I created the following content to prevent unnecessary churn when trying to create new innovative products. It describes a product development process I’ve refined over the years and use on a day-to-day basis when building compelling products customers love. The process is described in a way that will make it clear and easy to implement within your company. It is specifically designed for building large customer-facing features where “large” is defined as a product that requires 1 month or more of engineering time to complete.
Common Product Development Issues
First, it’s useful to point out the ways in which product development is typically broken or inefficient at young technology companies. Here are the common issues that I tend to see at startups:
The value you want to create for your customer has not been clearly articulated upfront.
Projects get “blown up” late in development due to large communication gaps during development.
Creating the first product prototype takes far too long, leading to a lull in the pace of development.
Customers aren’t being talked to enough, leading to products that don’t adequately reflect customer wants and needs.
The project team building the product doesn’t have a clear escalation path to get unblocked.
The below process has been designed to explicitly solve or greatly mitigate each of the above issues when developing new products.
In addition to solving common product development pitfalls, this method of developing products is rooted in a set of guiding principles which further prevents the above issues and gives product teams a common language to use when describing how they build product:
“Work backwards” from the customer: Start with intense focus and clarity on the value the company wants to create for customers as opposed to thinking about the value the company wants to create for itself. The belief is that if a startup makes the customer very satisfied, customers will engage deeper with the product, which leads to an increase in the key business metrics. Amazon is the best example of a company that begins product development with an intense focus on value to the customer.
Collaborative: All key functions (e.g. product, design, engineering, and customer support) are present from beginning to end since each function provides a unique and valuable perspective. That means everyone must own the outcome of the product— e.g. engineering should care just as much about the quality of the user experience as a designer should. I don’t believe in the “PM as the CEO of the product” idea because most PMs don’t have CEO quality judgement. Software development is best conducted as a team sport.
Interactive prototypes: A product development process should aim towards creating interactive prototypes worthy of being tested on actual customers, as quickly as possible. The reason is that startups learn the most when testing an interactive prototype on customers. Interactive can mean working code or a high fidelity visual prototype using something like Framer, which strings together visual designs through clickable hotspots.
Measure and learn: Once a product is shipped, you’ll want to measure the outcome to see if it created the expected impact. If not, you can investigate why that is the case and use those insights to either deprecate the product, improve it, or carry forward those learnings into future products that are built. Shipping products without understanding the impact is unacceptable.
A Repeatable Process for Innovation
First, I’ll describe the process. Following the description is a visual concept. The product development process follows these steps:
Begin with conducting Customer Research as part of “working backwards from the customer”. It’s through this research that you will refine the product hypotheses— i.e. what the product should do and why it should do it, what specific problems you’ll be solving for the customer, and what forms of delight you can provide. Customer Research can either be conducted by a PM or a designer, if your company doesn’t have a full-time research lead. Each conversation is 30 – 60 minutes and follows an open-ended format that allows for spontaneous discovery of rich customer insights. These insights should eventually make its way into product requirements.
In parallel, the lead Product Manager begins drafting product requirements (which also includes an Amazon-style press release). A draft of the product requirements and press release must be finished before starting the design sprint, which is how a product team develops its first testable prototype. The initial draft should be reviewed by the design and engineering leads, so they are familiar with it and can provide useful feedback. You want all key team members to be versed in what value you intend to create for the customer.
Once Customer Research is complete, and a first draft of product requirements and the press release have been drafted, the team will then run a design sprint to quickly design the first testable prototype of the product. I selected the Google Design sprint method since it was created with the time constraints of a technology company in mind. The issue with most traditional design processes is that they can take weeks to months to get to a testable prototype. That timeframe simply doesn’t work within a startup. The Google Design Sprint method is the most effective that I’ve seen when going from 0 to 1 within a software company. The design sprint takes 1 week, at most.
Once the design sprint is complete, the team can finalize the product requirements and Amazon-style press release so that the requirements and customer value are crystal clear before full development begins.
The results from customer research and the design sprint are brought into a kickoff meeting to get everyone on the same page prior to the development process ramping up to 100%. A kickoff meeting should be no longer than 45 minutes and should be conducted shortly after the design sprint is completed (e.g. within 1 week). You’ll want all primary decision-makers involved so that there are no surprises, which could lead to the project being derailed later in development. Feedback from primary stakeholders should then be taken into account and incorporated into the product plans.
Once development begins, the project team will present the latest prototype(s) (across all platforms— e.g. web, iOS, Android) and overall status of the project during weekly or bi-weekly product reviews until the product is finished and launched to the public. Product reviews are also 45 minutes max and should take significantly less time (e.g. 20 – 30 minutes), if run efficiently. The purpose of product reviews is to maintain coordination throughout the project, give the project team a regular interface with the leadership so that they can ask for help or support when needed, and to incorporate feedback on the prototypes iteratively.
This is a conceptual diagram for the product development process from start to finish. It’s very useful for project leads (especially the product manager) to have this process memorized, so that they always know what should be coming next in the development process. If run well, it should only take 2-3 weeks to finish customer research, the design sprint, and have a kickoff meeting session. Keep in mind that this is for new, innovative products/features, so getting to the point of alignment on a medium fidelity prototype is impressive in such a short timeframe. From there, development starts to move quickly until the product is ready to launch.
Here’s the full list of templates that you can used in conjunction with the process laid out above. This will allow you to incorporate some or all aspects of this process into your own team or company.
Thanks to an abundance of data storage, analysis, and visualization tools, startups today have the ability to make rapid improvements to nearly every aspect of their business. However, this overabundance has led to a significant bias in that startups now lean on structured data too much. So much so, in fact, that some of the fundamentals of building innovative products, such as rigorous customer development, have fallen by the wayside. One of the byproducts of this data obsession is that many startups try to optimize their way towards success through relentless A/B testing. This typically pulls them further away from essential insights and truths that they might discover, if they spent less time analyzing structured data from a database and more time collating the unstructured data that can be discovered when talking to customers.
The good news is that data over-reliance can be easily corrected with a shift in mindset and some of the tools and guides I provided in this four part series. In terms of next steps, I hope you take a few key steps from here. First, move forward with designing a company-wide org chart that creates an explicit balance between optimization efforts and innovation efforts. It’s also critical to make wise decisions with the types of experiments to run and avoid running tests that will never meaningfully improve your business. And finally, that you adopt some version of the repeatable product development process I shared, so that you can innovate much more effectively for the betterment of your customers and your business.
As reference, here are all posts in the series in case you’d like to read them again:
Each year, hundreds of millions of kids from around the world are introduced to sports. I was one of them starting back in 1988. My life was designed around sports starting at the age of six and continued, without interruption, until I took off to college when I was 18. This level of involvement meant that sports was more than a game I played on nights and weekends. It was my social circle for much of my life and it’s where I spent some of the best time with family too – playing alongside my brothers, having them cheer me on from the sidelines, or being coached by my dad. Sports also served as a purveyor of valuable life lessons, such as the value of teamwork, hard work, determination, and so on.
Despite the significant role that sports played in my life, however, I have almost nothing left from those years other than memories and a box of miscellaneous home run balls, trophies, and newspaper clippings buried in a box, tucked deeply into the corner of my closet, only to be revisited in a random bout of nostalgia once or twice a decade. No video footage. No letters from college scouts (because I was never “discovered”). And no way of sharing those moments with family members that didn’t have the chance to be there while I played.
That’s how things were when I played in the 80’s and 90’s – but with today’s technology, that doesn’t have to be the case anymore. Which leads me to the team at BallerTV and why I am keen to invest in this remarkable company.
BallerTV makes it easy for anyone to watch live streams of high school sports from a growing list of games across the country, whether it’s a family member who is unable to attend an away game or a college scout looking for talent in parts of the country they typically can’t cover. Not only can users watch games being broadcast live but they can access unlimited replays, download games to keep a digital memory to revisit, and player profiles that include highlight reels helping today’s up-and-coming players get discovered by other coaches and scouts.
I first met the founders Rob Angarita and Aaron Hawkey over coffee in the summer of 2018. They shared their passion for sports and their prior entrepreneurial history as co-founders of a product called Cramster, which was sold to Chegg and is now know as Chegg Study. They unpacked their vision for BallerTV and helped me understand the potential for the business and the underlying drivers they see shaping the industry that they are helping to create.
A new distribution method: whereas college sports benefited from mainstream cable television adoption, high school sports benefits from online services such as Youtube and now BallerTV. The first college sports game was broadcast on TV in 1939 but it wasn’t until mainstream adoption of cable television that college sports took off both in terms of viewership and revenue. As of 2017, the NCAA basketball tournament (better known as March Madness) generated $1B in television advertising revenue by itself. A similar ascent in viewership and revenue is starting with platforms like BallerTV. For example, players like Zion Williamson have amassed tens of millions of views of their high school highlight reels. He is just one of thousands of other amateur players to gain notoriety online before making the transition to college.
Lower cost game production: ESPN, CBS and other major networks only broadcast major amateur events (such as the McDonald’s All-American games) because they can only make the economics work for marquee events, given prohibitively high production costs. Turns out it is expensive to haul large television production equipment across the country and staff the event with highly paid on-air talent. In comparison, BallerTV is able to shoot amateur games at a fraction of the price as traditional broadcast television because of the proprietary technology they’ve created and ongoing advances in hardware, software, and communications technology.
Scalable organic growth: BallerTV has built an incredible brand within the amateur sports community. So much so that high school programs, tournaments, and leagues are banging down their door, asking BallerTV to record their games because parents and players find so much value in BallerTV’s products. For each new game that they record, more players, coaches, and scouts sign up so that they can get access to BallerTV’s fast-growing lineup of games. It’s so valuable that future Hall of Famer Dwayne Wade recently joined as brand ambassador for BallerTV.
When an investment opportunity gets to the finish line, I always do a gut check and ask myself, “Would I want to work here as a full-time employee?” In this case the answer was a clear “yes!” due to a constellation of factors. They have a small yet deeply committed team led by two co-founders with abilities in both execution and vision, a fast growing customer base that finds emotional value in the product they provide, and an increasingly large market that serves the greater good of helping kids grow up to be healthy, well-rounded adults. A company like this needs to exist and I’m very happy to be a part of it along the way.
Growth Hacking is declining in relevance. Will it disappear entirely? I don’t think so. Nor do I think it should. But the craze that once drove every startup (even enterprise!) to look for a Growth Hacker is on a steep decline. And I believe that’s a good thing.
How did it start?
In 2010 a very sharp technology marketer named Sean Ellis coined the term “Growth Hacking”. Andrew Chen, another very skilled technologist and current Partner at A16Z, followed on with a post describing the role of a Growth Hacker as the “new VP of marketing“. In parallel to that companies like Facebook and LinkedIn, which had two of the earliest and most successful growth teams, dating back to 2007/2008, received mainstream media attention for the efficacy of these little-known yet powerful Growth teams. In a very short period of time something was created out of nothing and the Growth Hackers phenomenon spread, as if by design, like wildfire throughout the technology industry. There are now several growth hacking bootcamps and large conferences. I’ve happily participated in some of them.
However, I observed an unhealthy interpretation and adoption of the growth mentality taking seed not too long after growth hacking became a thing. People spoke of growth hacking and hired for a growth hacker as if it were a panacea. Many months later they found themselves optimizing a product that consumers didn’t care for, dangerously short on cash, and with zero interested investors or buyers. There are a few examples of this happening in recent years. Viddy, once touted as the Instagram of video, amassed tens of millions of Monthly Active Users thanks to temporary prominence in Facebook’s news feed. They raised at a valuation in the hundreds of millions but shut down shortly after a flurry of growth and fundraising because it turned out they had lots of cheap, temporary distribution but very little user engagement because the product wasn’t useful.
By now I’m sure that hundreds or thousands of other startups have similarly discovered that a growth hacking mentality hasn’t led to the breakout moment they hoped for. That said, I would assume that interest in growth hacking has started to subside. It certainly feels like it. Fortunately, there’s data we can look at.
The search query data in Google Trends provides a simple proxy for the level of interest in Growth Hacking. Below is the global search query interest for the keyword “Growth Hacking” over the last several years. Worldwide query data reveals a healthy up and to the right trend, though with some observable deceleration over the last year in particular.
The story becomes a bit more interesting when you slice the query data by country. Here is the search query data in the United States. It’s down and to the right.
Interestingly, the next large market to adopt the term was India beginning in August of 2012. Strangely, it looks like the term is rebounding a bit in the last few months. I’m not sure why that’s the case yet it’s interesting to point out since the term is showing a consistent flattening or decline in most other large markets I looked at. Either way, this isn’t a healthy trend.
In the United Kingdom, lift off in searches for Growth Hacking didn’t begin until December of 2012. The query interest is steadily declining just as it is in the US.
Following that it spread to Germany by August of 2013. Germany hasn’t started its decline yet however search interest is may be flattening as of the last 2-3 months.
Shortly after that the term took hold in Brazil in October of 2013. The “market’ is still young relative to the US (the term took hold a little over a year later in Brazil) so I would expect flattening and a decline to kick in within the next 12 – 18 months.
Here’s the search query interest for all of these large markets in aggregate. This doesn’t look promising. Given that these large markets are down and to the right as a cohort, yet worldwide search interest is still on the rise, I’m assuming that worldwide intent is currently being driven by laggard, long-tail markets.
Nonetheless, I think it’s fair to say that we have data that validates the opening line of this blog post: growth hacking is declining in relevance.
Product Market Fit for Product Market Fit?
When I was writing this post I originally planned on solely looking at the search query data for “Growth Hacking”. Then it dawned on me (thank you Philz coffee?) to look at the search query data for the term “Product Market Fit” and to compare the two. I think this is a fascinating comparison because the assertion of Growth Hacking is that you can optimize your way towards scale while the assertion of Product Market Fit is that you must innovate your way towards scale.
First, let’s compare the relative search query interest worldwide between “growth hacking and “product market fit” from 2008 to present. I was shocked (mostly saddened) to see that query interest in growth hacking (the blue line) was many times greater than product market fit (the red line).
My gut reaction is that we’ve lost our damn minds if we think growth hacking is more important or compelling to research than product market fit. The generous interpretation would be that startup operators and founders understand product market fit much more than growth hacking, which leads to lower search interest in product market fit. That seems like a stretch so I’m not inclined to believe that interpretation.
Let’s slice it by country to see what else is going on.
Here’s India. Pretty uneventful other than to note the nearly non-existent growth for the term product market fit.
And in the United Kingdom the story is about the same.
Germany data tells a similar story to the UK, though with a smaller and more recent lift.
Lastly, here is Brazil. This makes me sad.
The data makes me think of a recent interview with Warren Buffet where he had a gem of a quote regarding why most investors don’t buy and hold a broadly diversified portfolio and instead choose to trade individual stocks in an attempt to outperform the market. When asked why most investors don’t follow his advice he remarked “Nobody wants to get rich slowly!” This behavioral phenomenon appears to be playing itself out in the startup world as well and I think this data provides reasonable support for that hypothesis. After all, who wants to build a billion dollar startup slowly?
But it’s not all doom and gloom. What if we looked at the term “product market fit” in isolation? The worldwide search intent data looks promising.
And here it is in the United States.
All other major markets I looked at show little or no growth in searches for “product market fit”. The UK is showing a bit of lift off in the last 1-2 years, but again, it’s minor relative to the search interest for growth hacking.
My hope is that the term Product Market Fit has Product Market Fit and that Growth Hacking continues its decline down to a more reasonable level. What that implies is that technologists, as a collective, will get back to our roots of building innovative products that people love, within great markets where better alternatives are needed, as opposed to optimizing our way towards vanity metrics.
The End of Growth Hacking and Growth Teams?
As I said at the beginning of this post, I don’t think Growth as a skill and a function should go away entirely. The concept of looking at data, running experiments, and optimizing products is valuable and necessary to consumer technology companies when applied in the right way, at the right time, and at the right types of companies. And in moderation! It is no replacement for innovation.
Yet I am glad that it’s going through its boom and bust cycle because that means we’ll move on from the hype and get down to the white hot center of what’s actually useful and relevant when it comes to building large, sustainable technology companies. The end result is that the best-in-breed of growth practitioners and bootcamps will remain. Reforge is an example of that. It is a world-class program put on by growth leaders that deeply understand the discipline. Importantly, they also understand that having a Growth team/focus is the side dish and not the main course.
A More Holistic Approach to Growth
Building a product and company that can grow sustainably takes much more than a few clever hacks. In future posts I’m going to spend a significant amount of time talking about the other elements of growth that receive very little attention and that deserve the spotlight. The list includes things like:
Running a process to get to Product Market Fit: if you don’t have a product that is valuable or necessary within a large and growing market, then nothing else matters.
Positioning effectively within your market: it’s important to consider what type of market you’re in, who your target customer is, and how to position your product within that market to compel your target customer to engage with your product.
Finding scaleable acquisition channels: assuming you establish Product Market Fit, you’ll want to open up the top of your funnel. I’ll provide practical guidance on how to identify new channels, test them, and scale them in a way that is economically efficient for your business. And, no, paid marketing isn’t the only solution.
Designing the ideal organization structure: most startups suck at designing an org and most significantly underestimating the importance of thoughtful org design. I’ll lay out my approach to creating org charts that align the company with what matters most, allows for nimble staffing changes in the event the company needs to shift in response to new information, and that strikes the appropriate balance between optimizing products versus innovating to build new products.
Knowing when to optimize versus innovate: growth can come from optimizing the existing products and services a company has, or from creating all new products and services that allows the company to deepen its relationship with existing customers or move into new market adjacencies. I’ll teach you how to decide which method to pursue and what balance to strike between the two. And how to reflect that in the organization you design as the company scales.
Being great at building innovative products: I’m shocked at how many consumer startups don’t set themselves up to continuously innovate on product. Being great at product innovation involves designing a product development process that can be leaned on to reliably produce high quality products at a fast pace. I’ll describe a process I’ve designed and taught other companies to implement.
Maintaining a pipeline of user/customer insights: in the words of Steve Blank, you have to “get out of the building” and continue to engage with your customer to discover rich insights which inform future innovation. I’ll write in detail about how to have productive customer conversations so you can do so within the walls of your own company.
My hope is that the collective attention of the consumer technology world will continue to shift away from growth hacking and more towards subjects that deserve a greater proportion of our attention.
I’m excited to announce that I’ll be joining Unusual Ventures as Partner, where I’ll be using my expertise in growth and product to invest in and scale the next generation of great consumer products.
Specifically, I’ll be focused on marketplaces (Airbnb/Uber), network effects businesses (Twitter/Facebook/LinkedIn), subscription services (Wealthfront), and content platforms (Quora), although I’m open to other types of consumer startups as long as the founders are building best-in-class products and services.
Fifteen years ago, I got my first job at a startup shortly after I finished college. I got that job by emailing the CEO directly. Thankfully, he saw something in me and gave me a chance. Since then, I’ve had the privilege of working on growth and product at a standout group of consumer technology companies such as Facebook, Twitter, Quora and Wealthfront — where I was most recently the President. You can’t anticipate where your career will take you, especially in startup land. It’s hard for me to express how appreciative I am of the unpredictable path that unfolded for me. That said, unpredictability can cut both ways.
In April of 2018, I had a sudden medical scare with my heart. I spent the night in the emergency room, and the following two months conducting a battery of tests to understand what was going on. The good news is that I’m now healthy and able to move forward with my life and career. The downside is that I had to step away from day-to-day operations at Wealthfront, a company that I love. Like the CEO of the first startup I joined, Wealthfront’s CEO, Andy Rachleff, decided to take a chance on me over five years ago — and he groomed me into the executive that I am today. I’m forever thankful to him and Wealthfront, and remain a long-term shareholder and staunch believer in the role that it will play in making sure that everyone has access to world-class financial products and services.
The time I spent shoring up my health also provided space for thinking through what I wanted to do next. It was during that time that the unpredictability of life cut back in a positive direction and I met John Vrionis and Jyoti Bansal of Unusual Ventures. I learned about the venture firm that they were building, the values underlying the firm, the vision they had for it, and the opportunity for someone like myself to play a role in building the consumer investing side of the firm.
A few specifics stood out to me the most. The first is that Unusual is changing the early stage investment model by working directly with entrepreneurs to do some of the early company-building work via their Get Ahead Platform. For example, if a portfolio company needs to hire engineers, Unusual’s in-house recruiters will do that for them. The firm also works with portfolio companies to develop their go-to-market strategies in addition to providing sales support to land their first customers — which is the hardest nut to crack for enterprise startups.
It’s not enough for today’s VCs to simply provide capital and occasional advice over email. This is a shared belief for everyone at Unusual. Now, I get to play a role in building out the hands-on services for consumer startups to match what we already have on the enterprise side. It’s an ideal setup for me because it means that I get to roll up my sleeves and work alongside entrepreneurs as they build and scale category-defining products — something I was fortunate enough to experience several times in my career. I never wanted to be an investor that solely provided access to money, so the fit with Unusual was a no-brainer.
All of these services are enabled by a team defined by servant leadership. It’s a value that Partner John Vrionis talks about a lot, and that I subscribed to even before I had the words for it. From a values perspective, Unusual is the most entrepreneur-obsessed firm I’ve encountered. Everyone at the firm works as a team rather than a collection of individuals. I love how much relationships matter in all of the work we do. The team cares just as much about one another as the founders we support. I saw an immediate match for my own values, and the behavior that I’ve tried to exhibit and reward throughout my career. .
One last thing that made me certain this was the right move: Unusual chooses to generate investment returns for people and institutions that don’t typically get to participate in Silicon Valley wealth creation. The firm’s LPs consist of nonprofits and historically black colleges, including Hampton College and Spelman College, as well as the United Negro College Fund. Diversity is not only being baked into the team from day one (50% of the team is female), but into the recipients of their investment returns as well. As someone who has always strived to work for inspirational purpose, I was quickly compelled, and I could see how my prior experiences were setting me up for this next step — to help lead a firm that makes purpose part of the every day.
I got a taste for the virtues of venture investing through advising and investing I did on my own time with startups such as Poshmark, Blue Bottle Coffee, and Opendoor — just to name a few. Watching passionate and talented people build new technologies is exciting, and assisting in their success by sharing the collective wisdom of all of my startup experiences is very fulfilling. Now, I can couple my knowledge of growth and product with significant capital to give entrepreneurs even more leverage to make their visions a reality.
As Steve Jobs once remarked, “You can’t connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something — your gut, destiny, life, karma, whatever.” Right now, those words seem especially insightful. I’m thrilled that I get to help the next generation of company builders, and even more so that I get to do it at a firm like Unusual.