Vanity Metrics and Other Nightmares with Metrics in Startups

Translated by Núria Adell.

One of the issues that gives us the most headaches at Nelio is that of metrics. As computer engineers, we love Lord Kelvie’s phrase: “what is not measured, cannot be improved”. So we put a great effort into analyzing as much information as possible from different perspectives, in order to make the decisions that will help us reach our strategic objectives.

But the harsh reality is that it’s complicated to get conclusive results that actually help us identify the road to success based on metrics. Next, we’ll talk about the KPIs that we try to manage and some of the difficulties with which we are dealing.

Key Metrics in a Startup

One of the first things you learn when creating a startup is the importance of key performance indicators (KPIs) that tell you how your business is doing. In your initial business plan you should set some objectives and define the indicators that will determine whether you reach them.

Oh, and allow me to take this opportunity to clarify the difference between the concept of metric and that of KPI, since they’re often interchangeably used, but are not exactly the same thing.

A business metric is a quantifiable measure that is used to calculate the progress or performance of a goal from a given perspective. For example, a metric is the number of monthly visits your webpage receives or the average duration of each visit, which can be obtained through Google Analytics.

KPIs are those metrics (as you can see, all KPIs are metrics, but not all metrics are KPIs) that are “key” to evaluate the performance of your business.

The metrics that are considered KPIs depend on the type of business you have. So, for example, the average duration of each visit will be considered a KPI in a news website, but not in an e-commerce one.

After this small clarification, let’s go back to where we were. I was saying that when you create your startup, you soon realize (if you didn’t already have the experience) that you need to create one or several spreadsheets with your KPIs. This will be your scorecard to improve your business. Of course, if you have left-over budget you can buy big tools that will do all the work for you.

But what KPIs do you need? Well, those that will help you answer the following questions:

• Is your business financially viable?
• What is working well and what needs improvement?
• What actions should be done to manage the business?
• Should you, as the CEO, step on the accelerator or the brake?
• What impact does stepping on the accelerator have on the cash and the income/expenses of your business?

Let’s see what these indicators are for a startup.

Key Sales and Customer Indicators for SaaS

If you opt for a SaaS business model, you’re like us. In our case, we have Nelio A/B Testing and Nelio Content, which work by monthly or annual subscription.

The article SaaS Metrics 2.0 – A Guide to Measuring and Improving What Matters by David Skok gives you a more complete and detailed vision of the key indicators that you need to understand and optimize in a business with these characteristics.

According to Skok the two key indicators that you should get in a new SaaS business are:

On the one hand, Skok tells us that the value of any business generated by a normal customer (Lifetime Value, LTV) should be more than three times the customer acquisition cost (CAC). For more information on how to calculate these two values click here.

On the other hand, to know whether your company is profitable, you should be able to recover the acquisition cost in less than 12 months. As you can see, it makes sense to think that if you achieve these two KPIs, you’re on the right track.

I would also like to add that in Skok’s article you will find a lot of information about the importance of another metric directly related to the LTV: the Churn rate or abandonment rate of a customer. You’ll find more details about this in the previous link.

Nelio A/B Testing

I was very impressed by the quality of this plugin, how easy it was to set up, and the outstanding support Nelio provides. I highly recommend Nelio A/B Testing.

Josette Millar

WordPress.org

View Plans

Key Liquidity Indicator: Cash Burn Rate

Liquid assets are key in every company. So if there’s a number that you have to control very well from the start, that’s the one that tells us how much money we “burn” monthly (Cash Burn Rate or CBR). This is the indicator of the speed at which we consume our funds, including rents, salaries, marketing, services, etc.

This doesn’t only give us the cost of functioning, but also, if we divide it by the money we currently have available, it tells us in months what we have left until we run out of money, if we didn’t get any additional income. It can help us decide when we have to start reducing costs, looking for external financing, or break even in the business, when costs are covered by the revenue.

Key Indicators for a Website

If your business is online and the income comes from the website, you can get the indicators that will provide you with information about how your business is doing directly from the website. But which ones are key?

The KPIs for a website that Avinash Kaushik recommends you have are the following:

• Conversion Rate: the percentage of visitors to your website that do what you want them to do (buy a product, subscribe to your newsletter, etc.).
• Average order value: when a user buys something on your website, what’s the average quantity that is spent?
• Time to purchase: does a buyer make the purchase on his first visit to the website, or has it been days or months until he makes the purchasing decision?
• Visitor loyalty: how often and/or with what recurrence do your visitors return?
• Task completion rate: what percentage of the visitors to your website that have a goal actually reach it?
• Share of search: what volume of all searches with your keywords reaches your website?

Here you have a very interesting list of metrics that can be of great help in making decisions to be successful with your website.

You just have to create a spreadsheet with all of them, make the right decisions according to the values you get, and your business will grow like foam.

What a pity that everything always has to be more complicated than we’d like… what difficulties and problems have we encountered ourselves?

Problems with the KPIs

#1 How Can I Define Them Accurately and Rigorously?

When I started calculating Skok’s metrics, I encountered the following problem: calculating the monthly recurring revenue (MRR) when you have monthly and yearly plans, along with changes in plans by users, is not easy to capture. With Nelio A/B Testing, we’ve even had customers asking us to keep their subscription in stand-by in order to resume it later. How do I keep this in mind when calculating the MRR?

On the other hand, the CAC is the sum of all sales and marketing expenses divided by the number of new customers. But what exactly do I include in the concept of “sales and marketing expenses”? The money I spend on AdWords and other paid marketing campaigns only? Do I include the salaries of those who engage in marketing? And what about the salaries of the support team during the trial period of a product? Each support ticket is a marketing opportunity, right?

And you’ll find similar problems with the other metrics: being accurate and rigorous is not that easy.

#2 How Do I Get the Information to Calculate Them?

Another problem we’ve encountered is how complicated it can be to obtain the information to calculate certain metrics.

For example, let’s look at the indicators of a website. The top four mentioned are metrics that you can define in Google Analytics. But let’s face it, learning the basics of Google Analytics is not trivial, and becoming a Google Analytics expert is another whole level. I recommend you to start with The Absolute Beginner’s Guide to Google Analytics by Kristi Hines.

Then, you can consider getting the Google Analytics certificate, which although most people get it by studying online, the fact that there are 6-week courses offered to get the title shows that you don’t just get it by reading a couple of articles.

Another example: the SEO metrics of your keywords can be manually calculated by looking at the number of global searches of those keywords with Google Trends and analyzing the visits to your website with Google Analytics. And although you have very complete tools that can help you perform these tasks, you can see that it’s not automatic either.

That is, you can invest in training or experts and tools that can help you out, but in any case, it’s not that easy to implement a method that automatically gets you all this information of your website. And the problem doesn’t end here.

#3 What If I Can’t Get the Information?

It can still get more complicated. If you have multiple sales channels for your product, getting information can become impossible. For example, in our case, a percentage of users of our products have known us through the Directory of WordPress plugins and have never visited our website.

As we don’t have access to the Google Analytics of our plugins in the WordPress directory, the information we have about that traffic is very limited (if not non-existent).

Therefore, we find ourselves in a situation in which the analytical information that we obtain from our main sales channel is very scarce. Is it right to make many decisions based on the information we get from our website even if it’s not our main sales channel?

#4 How Do We Monitor the Impact of Changes?

Another recurring problem on the web and marketing is that it’s very difficult to monitor the real impact of any change and analyze it to draw conclusions and start over. Why? Because we never perform totally isolated actions. At all times, we’re carrying out multiple actions to improve any aspect of the business.

For example, at a given time, focusing on improving the SEO we changed some aspects of the website to reduce the loading time of the pages. We also removed very old items from our server that were obsolete. Finally, we installed the plugin MultilingualPress to better manage the translations between the English and the Spanish versions of the WordPress multisite that is our website.

The traffic then grew and continues to grow, but unfortunately, we’re not able to demonstrate whether such growth is due to one or more of the changes we made, or because Google likes our articles more now, or because of any other changes we had previously implemented.

It’s not that we’re doing actions without thinking about them. In fact, the actions we’re carrying out are always based on articles written by great experts in the field (I take off my hat to the great SEO experts). But as you can see, the theory is always great, but in practice, being able to know what you’ve achieved with each action is complicated.

Therefore, precisely because of the complexity of the whole process, there is a tendency to try to simplify our lives by calculating other metrics that can help us without taking up too much time.

#5 Vanity Metrics

And this is where we get to vanity metrics.

Reports on your finances, Google Analytics, Google Adwords, and social networks, can offer you a set of metrics that are easy to measure or monitor, so you don’t have to complicate your life as much as you do with the previous ones.

For example, the annual revenue accrued, the number of visits to a web page, the number of followers on Twitter, the number of Facebook fans, or the number of downloads of an application. These metrics give you information that you think is useful and visible, but you’ll find a couple of problems with them.

The first big problem is that they give a false impression of growth. It’s clear that any accumulated metric will always grow. The line shown in these graphs always portrays a fantastic growth that you’d never get tired of looking at. The second problem is that KPIs need to be metrics that help you make strategic decisions. But the reality is that there are metrics that are not very useful. They’re not actionable metrics. They may be useful to “motivate or deceive” some potential employee or investor, but not for much more.

So it’s probably not such a good idea to simplify the calculation of metrics…

#6 Are They Actually Helpful at the Beginning?

Going back to the indicators discussed above, at the beginning, when your new product has only been in the market for a couple of months, you have to question them fully.

For example, the LTV of a monthly subscriber at that time can’t be greater than the equivalent of two months paying for a subscription (your product is that new, right?). This figure doesn’t reflect the concept of LTV that can be useful to make decisions. Likewise, making your product known at the beginning implies a titanic effort, so you can assume a CAC that doesn’t reflect the actual CAC under conditions of “cruise speed”.

The first phases of introducing a product to the market, even when following a methodology of Lean Startup, are quite different to the phases in which your product has been marketed for some time.

Should we actually make decisions based on LTV and CAC during the first few months?

Some Conclusions

So far, we’ve seen some of the problems of the metrics. We spend a lot of time researching and trying to get the indicators that will help us make the most appropriate business decisions at all times. In short, here are some of the conclusions we’ve reached.

The usefulness of the metrics we analyze is less than what we’d like. In fact, those that have clearly been and continue to be actionable for us are those obtained from A/B tests with Nelio A/B Testing to improve the conversion of our website.

How do we make decisions to improve our goals? The truth is that some decisions are based on expert recommendations. For some others, such as deciding whether to invest in Adwords, we perform a test to compare the cost of carrying out that action and the time or results needed to recover that cost. If the result doesn’t seem reasonable, we don’t repeat the process.

But some other actions, such as our participation in a WordCamp Europe, are done because we like and love participating in the WordPress community. We could justify many of the benefits they bring us, but surely we wouldn’t be able to quantify the economic impact it has on our company.

And indeed, although we’re not always able to measure everything, we carry on trying. We’re still convinced that the 4 pillars of success in our startup are those described here.

So my question is the following. Are we the only startup that encounters these difficulties? We’d love to hear about your experience and knowledge to proof that we’re not alone in the desert.

Featured image by Jonna Fransa