A no-show prediction model that looked great — until I found it was reading the future

At a health-tech startup, my project was a model to predict which patients would no-show for appointments so the clinic could send extra reminders. My first version had an AUC around 0.94, which honestly felt too good for a summer intern. When I looked at feature importance, one feature dominated: an appointment 'status' field. It turned out that field got updated after the visit, so it already encoded whether the person showed up. Classic leakage, the model was reading the answer. I dropped every feature that was written at or after appointment time and rebuilt using only things known when the appointment was booked: lead time, prior no-show count, day of week, distance bucket, appointment type. AUC fell to about 0.71. Less impressive, but real. I picked a probability threshold by looking at what reminder capacity the front desk actually had, not by maximizing accuracy, since the classes were imbalanced. I wrote up the leakage story explicitly in my final doc because I wanted the next person to not trust that status field. The clinic ran a small pilot sending reminders to the top-risk bucket. I framed it as a promising signal, not a solved problem.