I spent a few weeks on-and-off between more important study, trying to help a friend process their sequencing data in a bespoke way. I kept trying to write an algorithm that would take about 500 hours to run. I first used Pandas (which is kinda slow) and then tried base python dict() and iterables. Eventually I realised that most of the data was actually redundant if I just counted how many instances of each unique row were in my dataset and then just threw away the extras while keeping the counts for later. My new algorithm did what I wanted in 8 seconds flat and was only a tiny bit more effort to integrate with the counts.

My lesson learnt was that I just needed to reduce how much data I was working with first, instead of trying to stuff a multi-gb file into memory!