> Disappointingly, I ended up writing my data to a text file, sorting with unix sort (w/ some tuning on --parallel and --buffer-size) and reading it back.

One of my biggest beefs with Unix is that so much powerful functionality is locked up in command line utilities and not accessible as libraries.