How to Avoid Survivorship Bias in Your Search for the Perfect Customer

Running a Saas business can sometimes feel like a series of quests. Find a problem to solve. Build the solution. Find your first customers, etc.

Often times, these quests arise as problem-solving expeditions during our journey toward company success. A few weeks back, Josh Pigford of Baremetrics, a leader in the Saas analytics space, reflected on a time where his company’s dwindling cash reserves prompted his quest for the perfect customer. The concept is straightforward: talk to your best customers, find out who they are, and focus on finding more of them.

The Baremetrics founder suggests that a company with at least 100 customers should go through this process to successfully scale. Finding your perfect customer helps you “stop wasting time” on people who aren’t a good fit, Pigford reflected in his recent Founder’s Journey blog post. Find your perfect customer, and you will know where to spend marketing dollars, how to lower your support load, and accelerate sales cycles. Finding that perfect customer, Pigford argues, should also result in less churn.

Here at Raaft, we agree that the quest for the perfect customer should focus on understanding your paying customers, with a twist. We think finding the perfect customer means understanding all of your customers, especially the ones that cancel. If your search for the perfect customer ignores your canceling customers, you may unintentionally find yourself a victim of survivorship bias, and it may prevent your company from finding your best customer.

Dave Kellog, CEO of Host Analytics and a speaker at Saastr earlier this year, is an early and eloquent adopter of the problem of survivorship bias for Saas companies. To borrow his best example of survivorship bias, consider the following hypothetical from World War II: On the Western Front, military leaders were evaluating Allied air forces and trying to figure out what separated planes that made it back from a dogfight from those that did not. Planes that returned from battle had certain bullet mark patterns. The military powers deduced that they should put more armor on the areas where these bullet holes were located so that the planes that crashed would have a better chance of surviving. In fact, the opposite was true: the planes that were shot down had bullet holes in the exact opposite places of the surviving planes. The armor needed to go on those locations, not in the places where the surviving planes were shot up.

The military top brass made decisions on where to add plane armor based on the what they could see, when it was what they couldn’t see that held the right answers. That’s the essence of survivorship bias.

To avoid survivorship bias, “as-a-service” companies need to talk to their canceling customers as well as their paying customers in order to identify what characteristics or traits differentiate perfect from (possibly) worst. Who knows, the exercise may even help your company expand its definition of perfect customer to include a new vertical.

Survivorship Bias Case Study: Wine-As-A-Service

Let’s say you sell wine-as-a-service through a monthly delivery club. You have been in business for two years. MRR is declining and your churn rate is rising.

You send a survey out to your paying customers with the highest MRR and the longest subscription period.

The surveyed customers have a few traits in common:

Survey of “Best Customers”

Location: Each Coast
Annual Income: $125,000-$150,000
Education: Master’s or higher
Media: Wine Spectator (print version)
Monthly budget for wine: $100

Armed with this information, you focus your content, marketing and onboarding resources to target and retain these potential perfect customers. Your social media buys are on the East Coast primarily, and you pay a premium to be in high profile markets like Boston and New York. You buy expensive advertising space in the print version of Wine Spectator. You adjust your advertising filter to exclude people below the income level but pick income up to $300,000, and price your subscription boxes to be at or below $100 per month.

Let’s play out the same hypothetical with churned customers, and adjust our perspectives.

Survey of Churned Customers

Location: urban areas
Annual Income: greater than $200,000
Education: Master’s or higher
Media: None. Referrals/recommendations only
Monthly budget for wine: $225

Before talking to its churned customers, the hypothetical wine subscription company was wasting money on advertising in expensive urban markets and upper echelon income brackets.

However, by comparing the data gathered from best customers and comparing it to churned customers, the company has more accurate information it can test and further clarify:

The best customers might not be people living on the East Coast, but rather in suburban or rural areas who have less access to large wine shops or a variety of wine
People who are willing to pay for a print subscription to Wine Spectator might be a target market
Income is not a differentiating factor per se, and requires additional information. High-income earners might not be a target market.
Age may be a differentiating factor, based on geographic location and preference for media
Conventional advertising buys in print media may be more successful than social media campaigns.
To access high-income urban customers, the company may need to explore new products or delivery methods.

Without the comparative information from churned customers, none of the assumptions about a company’s best customers are completely accurate. The consequences for the wine-as-a-service company could be devastating: wasted money on urban, East Coast markets. Paying too much money on social media buys to target upper income clients beyond the perfect customer range.

In the end, eliminating survivorship bias is all about visibility. You don’t know what you cannot see. And if you leave churned customers out of your quest for the perfect customer, you are voluntarily closing one eye.

To discover how Raaft can lower your Churn for your Saas, book a demo from Raaft today!