An A/B test I was told 'won' — until I checked whether it was even significant

My first real project at a fintech was writing the readout for an onboarding A/B test. The variant had a 4.5% completion rate versus 4.1% for control, and the growth lead was already talking about rolling it out. When I actually ran the numbers, the test had about 1,800 users per arm, and that lift was not statistically significant, the confidence interval on the difference comfortably crossed zero. I also noticed the experiment had only run four days, so it was skewed toward weekday users and one arm caught a marketing email blast the other did not. I did a two-proportion z-test, reported the p-value and interval, and did a rough power calculation showing we would need several times more users to reliably detect a lift that small. This was awkward because people wanted a yes. I did not say 'the variant is worse,' I said 'we cannot conclude it is better yet, and here is the sample size to get there.' We ended up extending the test to two full weeks. It eventually landed as a real but smaller lift. The lesson I took was that my job was to protect the decision from a false positive, even when everyone in the room wanted the win.