As someone who’s also been part of multiple startups that scaled, I understand the challenge. The testing toolkit will depend on the nature of your product, and how substantial the risks or promised benefits. For many startups that are just trying to find their market, this can be doubly challenging. In the case of your startup Known, for example, it wasn’t immediately apparent that education would be an important early market. Nor were the risks or promises particularly high. That’s why we included a factor for scale.
Were Known to double down on the educational market, you would gain a rich tradition of evaluation that already exists: Books like By Design: Planning Research on Higher Education offer helpful introductions to how to do responsible evaluation of educational initiatives.
I agree that teams often fall short of their desire to do the right thing. I see that as a lack of resources and imagination. At present, I get the sense that fears of PR risk tip the cost/benefit against public research. By writing this article and working on CivilServant, we hope to reduce the costs and showcase the benefits, shifting the balance toward public interest research.
But knowing when and how to test is hard; that’s why we talk about supporting an ecosystem that spreads collective knowledge about the methods and ethics around these questions.