Testing, testing: On experimental drift and data driven product design
Yizhar (Izzy) Toren
A/B testing is (and should be) the gold standard for making data driven decisions. However, basing your decisions solely on tests can lead to very bad product decisions, primarily because of different types of hard-to-track changes to your environment (aka "experimental drift"). In this talk I will explain what experimental drift is and how it can affect your product design and A/B testing choices. I will also review a few strategies of handling drift as a data scientist working in a product team and show examples.