Every department has a KPI. For sales, it’s revenue. For engineering, velocity. Customer success has NPS, marketing has MQL. But what about product? What metric measures product’s effectiveness and impact?
Product leaders don’t have a primary KPI because they don’t have a singular metric to measure effectiveness.
If you measure your product’s success in more than just monetary value, you need to replace ROI with MoAR for effective product portfolio prioritization.
Why? Because Metrics on Available Resources (MoAR) is a more immediate measure than Return on Investment (ROI) when evaluating opportunities that maximize product and business outcomes.
The shift from ROI
Portfolio Management is a financial concept – allocate investments in different asset classes based on assumed risk and reward to optimize the overall outcome. It also ensures that the organization spends the most scarce resources on the work that yields the most value.
Years ago, I led strategic planning for our product portfolios across PayPal. We needed to evaluate the potential opportunities that may help the company achieve its several objectives. This ranged from improving engagement, launching into a new market, reducing transaction costs, as well as taking risky-bets on emerging trends.
How could we compare these diverse types of product initiatives across revenue growth, cost reduction, and risky innovation?– CFO and Head of Product
We started using ROI as most companies do but quickly ran into many limitations. We needed a solution to measure and evaluate opportunities more directly than ROI. Subsequently, this led us to develop a new approach – MoAR, or Metric on Available Resources.
First, let’s take a look at what ROI means
ROI, Return on Investment, is a ratio between net-profit (over a period) and the cost of investment. ROI is a performance measure often used to evaluate or compare the efficiency of many different investments. (Wikipedia)
ROI works well in traditional industries or functions where benefits and scarce resources can easily be measured in money.
However, ROI does not work well for companies that need to measure outcome in factors beyond money, or have other forms of scarce resources.
Product Portfolio Metrics: Measuring benefits
In the world of manufactured products, measuring the financial impact of a new product is relatively clear. If a newly designed cup is built and sold, we can measure how much revenue it generates in 1-year and figure out the total costs to calculate it’s 1-year ROI.
In contrast, a software product consists of many features. A proposed new feature may have a relative contribution to a business metric, which may be used to infer a monetary benefit. Even so, it is nearly impossible to directly measure and attribute the financial outcome of individual product features.
Let’s take a look at an example.
Based on past experience, we know that increasing sign-up conversion by 5% will result in $100k in quarterly profit if everything else remains unchanged. Assume Feature A aims to increase sign-up by 2%. Thus, we can infer that a 2% improvement would result in a $40k profit.
At the end of the quarter, we can measure the change in profit, but it is impossible to attribute the profit change to a given feature.
Therefore, using the financial impact of a product feature does not provide feedback on the benefit of a feature. But, measuring the sign-up conversion ratio over time can be easily achieved.
A monetary-based benefit measure also has a negative bias against innovation and risk-taking. The same amount of investment on the cash-cow mature business always generates a much higher money-return than investing in nascent products.
In the podcast “What Really Happened When E-bay Went To China”, Alan Tien, the former GM of PayPal China, highlighted how difficult it was to justify R&D investment to improve product in a smaller (China!) market. The full podcast is here (Minute 4:30).
On measuring cost (of sacred resources)
Particularly in technology companies, money is often not the only scarce resource. The most limited resources is in fact, human capital. For example, currently, there are more than half-a-million unfilled engineering job-openings. $1 million does not equate to the four senior Python engineers the company now needs to build the product and achieve its growth metrics.
So, what is MoAR?
The Metrics on Available Resources (“MoAR”) is the ratio of relevant contribution of a product feature towards a metric used to measure a business outcome, over the number of resources needed to achieve this contribution.
MoAR enables a more direct and holistic measure for portfolio planning and a clearer visual of product portfolio metrics.
How does MoAR fit product planning?
Start with defining the appropriate metrics to measure benefits towards the desired business outcome or objectives.
Metrics e.g., user growth, referral rate, retention, NPS, are the direct measure of product benefits. They are often leading indicators of monetary-based returns. Furthermore, companies that adopt an OKR or V2MOM framework can easily connect outcome targets with product portfolio roadmapping. MoAR enables an outcome-driven portfolio roadmap.
Available Resources are a better indicator of scarce resources. Examples include the Scala Engineers or the UX designer who designed the last app.
During portfolio roadmap planning, compare features that require the same resources using MoAR to prioritize. Add up the contribution of all planned features towards the goal. The total contribution should add up to 100% or higher to ensure the goal is achievable based on the current plan, with the resources available. Furthermore, this exercise exposes any known outcome gap and/or resource gap early on.
For example, assume a key business goal for the next quarter is to increase conversion by 10%. Three features are coming to the team and there are only 16 weeks of front-end resources available.
- Feature A is expected to contribute to 40% of this goal and requires eight weeks of front-end resources. The MoAR for Feature A is 40/8 = 5
- Feature B may add to 20% of this goal and requires eight weeks of front-end resources. The MoAR for Feature B is 20/8 = 2.5
- Feature C contributes to a cost-saving metric, but not towards the conversion goal, and requires four weeks of front end resources. Thus, the MoAR for Feature C is 0.
|Feature A||Feature B||Feature C||Total|
Let’s evaluate these features
First, we notice the shortage of front end resources required to support these three features. We will de-prioritize Feature C as it’s MoAR is 0.
However, this plan is not ready to go yet because you can see there is an outcome gap – Feature A and B will contribute to only 60% of the target metric.
The team needs to either reduce the target outcome by 40% and get stakeholder buy-in or start over to design a different feature set to achieve 100% of the target.
Identifying these gaps during the planning phase gives the team more time to address them before it’s too late.
The Metrics over Available Resources (“MoAR”) prioritization method allows companies to readily incorporate a direct measure against goals into their roadmap planning/ dynamic alignment process. This enables better product decisions, easily visualized product portfolio metrics, and improved overall outcomes.
MoAR is a key element of Responsive PPM – the new portfolio management approach for outcome-driven organizations.
* * *
Want to build products that achieve both near-term results and long-term vision? Sign up for a 14-day free trial of Dragonboat.
Dragonboat is the #1 Responsive Portfolio Platform connecting goals, strategies, and resources and seamlessly integrated with Jira, Github, and Clubhouse.