I scraped 1.75M WWI/WWII soldier records and built an infinite scroll memorial [OC]

For Remembrance Day, I spent 72 hours building theywerehere.co.uk – a searchable database of every Commonwealth soldier who died in WWI and WWII.

The Data

• Source: Commonwealth War Graves Commission
• Records: 1,750,608 soldiers
• Fields: Name, rank, regiment, date, cemetery, age

The Tech

• Scraped with TypeScript + Puppeteer
• Postgres on Supabase
• Next.js frontend
• Infinite scroll with virtual windowing

Why I built it

My great-grandfather's name is somewhere in those 1.75M. So I built this so no soldier is just a statistic.

theywerehere.co.uk

Happy to answer technical questions about the scraping/database/UI choices.