Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This is time partitioning, not indexing, but it's commonly done in data lakes for raw telemetry/log data.

    mydata/
        2019/
            11/
               12/
                   2019111200.csv
                   2019111201.csv

Where each one of those CSV files could be 10-100 GB in size.

Usually you want to process it into a columnar format like Parquet from there, though



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: