Listening to Software Engineering Daily, I heard that the there is a trend toward developing 'data engineering' as a discipline that handles sanitizing the data pipeline so that data scientists can spend more time working at business abstraction layers.
There's probably a scale at which that works better, two pizza data science teams for example.
Yeah, DE seems to have emerged as the title for those who manage large databases -- who regularly import raw data, cleanse, normalize, update, construct analysis pipelines, scale up and parallelize processes, and validate results. It lies somewhere between a DBA, sysadmin, and HPC engineer, with an awareness of basic stats -- where mastery of Hadoop's many components might converge.
It seems like a natural evolution of DB admin for very large scale noSQL and DBs, esp. those with unstructured data and often non-commodity architecture. I've seen numerous companies looking for such folks and I suspect demand will rise.
IMO, it's not a job you'd want to outsource. The role is too mission critical, the skills not predictable enough to be a commodity, and the penalty for screwing up is too great.
My issue with these sanitizing pipelines is that they only really work for fields where the data generation is relatively stable. Meaning, whatever instrument/method used in generating the data doesn't change dramatically every year. It's extremely challenging to design a all-purpose sanitizing pipeline.
So I can imagine a pipeline developed for use with well established social media APIs or standard scientific experiments that have been in used for decades. But it is hard to imagine a pipeline that can handle amorphous emerging high throughput instruments/methods.
There's probably a scale at which that works better, two pizza data science teams for example.