2014 Startup Digest SF/SV July user survey analysis

David Kim and Peter Shin

david.kim@startupdigestmail.com

peter.shin@startupdigestmail.com

Summary

• Learn to ask better questions

– Win: we have a high net promoter score (8)!

– Loss: we do not have clear drivers of success

• Of 96 unique responses:

– 30% were Founders, VCs, or C-level readers

– 5% were interns/students

• We should focus on success metrics that matter

Net promotion score distribution

-

5

10

15

20

25

30

35

3 4 5 6 7 8 9 10

# re

spo

nse

s

score

distribution of net promotion scores

Cross-section of responses

But What Factors Drive NPS?

• And are those factors statistically significant?

• Hypothesis (very limited given the data set we collected – see appendix for the survey): – Organizers promote SD more highly

– Years lived in the area is also positively correlated to score

• Or mathematically: – NPS = a*(Organizer) + b*(Yrs lived) + intercept

– Where Organizer is a dummy variable (i.e. 0 or 1)

Regression results

• Surprisingly, longer you live here, less likely you are to recommend the digest to friends. Call it cynicism

• Sadly, neither variables are statistically significant

– specifically, t-stat not above 2 (or p-value not low enough)

• Moreover, adjusted R^2 suggests that these factors basically have no explanatory power of the NPS

Conclusion & Afterthoughts

• In conclusion, we know we have a very high net promoter score (8)

– But, we don’t know why. We did not design our survey with the intent of discovering key metric drivers, but that would have been nice

• 100 responses among tens of thousands of readers is rather small

– Key takeaway is we did not have a baseline starting out

– Now we have one, and we can improve upon it

Afterthoughts, cont.

• We suspect that NPS is a vanity metric

• The real measures that matter are:

– Reach (equivalent to revenues for a business, and reflects sharing)

– Click-through rates (reflects usefulness to users, and therefore reflects quality perception)

• In future surveys or partnership initiatives, we have the above baseline, and can optimize efforts that increase those outcomes

Appendix: data clean-up

• Original survey link: – https://docs.google.com/forms/d/19k2gXjkDmplPo3n7BsK

v1VgrmlP-UvSADLnABYTr5Cc/viewform

• Data clean-up notes – Organizer column: 0 = no, 1 = yes

– Yrs lived: reduced <1 yr and n/a to 0

– Job position: created new categories to reflect responses as closely as possible

Appendix: data tables