Last week, we wrote about whether your startup is focused on discovery and learning, rather than on business plan execution. We referenced a local startup that has significantly changed its UI four times in four months. We know that the founders of this startup are smart people, and they were probably changing aspects of their landing page UI in order to capture more metrics, learn about their users, and test the ability of various distribution channels to scale. At least, that is what we’re going to assume.
What type of information is management trying to capture with all those significant UI changes? We could produce a “laundry list” of items – which everyone should recognize since you can find these metrics in dozens of blogs: logging visitors via which links, how does registration change when people are pushed to one landing page over another, did the users take advantage of a specific offer on one page over another, how do you get users to return, etc., etc. etc.
Simply stated, the real goal of the building a software product is to acquire customers – and do so as quickly as possible while you are obtaining customer feedback and improving your product. Period.
The field of Applied Econometrics (our academic field of specialization) is focused on multi-variate analysis of data through hypothesis testing of equations and their respective components. Early on, we learned that a hypothesis test must be very well defined in order to generate a sufficiently interesting response. What most people don’t realize is that “null” response provides as much information as a positive response. In other words, either you didn’t define your hypothesis test correctly in the beginning, or that null response is actually telling you something that should lead to an action. When the hypothesis test is not correctly defined, one will generate an indeterminate response. It’s the analysis of the indeterminate response that leads to better hypothesis testing, and it’s that analysis that straddles the line between art and science.
The question we would ask: Is there a “method to the madness” of running tests and iterating? What the metrics you need to have beyond the obvious ones above? What is the plan for the tests, and how does the testing program drive customer acquisition, and hence, drive revenue?
- What are the expectations for any test before the tests are run?
- Did the test achieve 10% of expectations, or 90% of expectations?
- If only 10% was achieved, then what was learned that would lead to a minor tweak, or should one consider moving on to the other plan?
- How are you defining your tests to determine which marketing/distribution channels drive viral customers?
- What was the profit margin of the customers gained from one method? What did you expect prior to testing? Was it high or low?
- How do you drive viral marketing once you get customers and traction?
- Can you continue to deliver profitable customers as you expand your customer base?
- What was your expectation of profitability as you created this test?
- How do the tests connect back to the financial statements beyond revenue generation?
- Was the implementation successful in generating customers delivering a high margin result, or a low margin result?
- In other words, what was the cost of the revenue to profits?
- Were the tests designed to take advantage of the viral marketing on mobile platforms?
All of these concepts revolve around multidimensional analysis, and the ability to reduce the multiple millions of combinations, to the few that drive a successful testing program since the key to building a long term business, is building a profitable business.
Or….you can do what everyone else does, and learn by the seat of your pants – as you burn through your cash.
Richard Piotrowski CFA, is the Managing Partner of Outram Research LLC, which focuses on assisting startups to improve their funding pitches to the Angel and VC investors. You can follow Richard on Twitter: @Angelpitchdoc. He can be reached at firstname.lastname@example.org, or at his blog: angelpitchdoc.wordpress.com. Also check out our website: www.outramresearch.com