On the other hand, agile is the anathema of good data modelling and it causes productivity gains early on which lead to ossification and inflexibility later
If you stop refactoring. So you know what? Don't stop.
At the beginning is the project when you know the least. About your tech, about your product, about your customers, about your competitors and partners. The best design decisions are made with the most information. Ergo, all design decisions should be delayed as long as possible.
The only question is how we responsibly delay design decisions. The answer for me is lots of automated tests and a willingness to refactor as we see ways to improve our designs. Plus a bunch of other agile technical practices.
This works only if you use an RDBMS (instead of a NoSQL solution) and you use it as a very dumb data store. For large classes of applications you lose far more than you gain. The reason is that you are FORCED to refactor your data with an RDBMS.
As soon as you move outside the RDMBS storage model with agile development something very icky happens. Old data structures stick around until actively cleaned up using processes which may take a long time to complete. If you have half a billion documents in Mongo DB created and used by your agile app, you are going to have to have your application handle those data-structure-wise forever. Every new change then creates an additional corner case that must be perpetually handled.
I don't get why you think NoSQL solutions are exempt from refactoring and maintaining data quality. I've happily done that with non-SQL stores.
No matter what development approach you use, you will eventually have to deal with change. No interesting application hits v 1.0 and sticks there. The only question is whether you embrace change, which is the Agile approach, or whether you resist it. If you resist it, you change in big lurches when circumstances force it. Having tried both, I favor the former regardless of what storage mechanism I'm using.
Refactoring your data structures is different though, is it not? In an RDBMS, you have to migrate your data when the schema changes, correct?
If you have, say, 100 million records in Mongo DB and you decide you need to refactor your data schemas, how long will it take you to do that?
The problem is that an RDBMS is very rigid typically on data input but very flexible on data output. if you change data input you MUST migrate data with it. MongoDB and others are very flexible with data input but what you get out is more or less what you have told it to store. This is a very different problem. You lose the ability to continue to refactor your data structure semantics beyond a certain point. This is not really the case with an RDBMS.
BTW, I have started to get PostgreSQL object-relational modelling and it is amazing. I suspect that although the barrier to entry (due to knowledge requried) is high, multiple table inheritance ought to allow far more agile development of database schemata than standard relational models allow, and at the same time do full intelligent database stuff.
100 million records may take a lot of time to update, but so what? It's not like you make major schema changes to the same data every day. If you write your code to read either format v1 or v2 and to output v2, the migration can begin gradually. When you're sure it's working well, you start a low-priority job that reads old records and migrates them. Eventually, all data is in v2, so you can ditch the reader code for v1.
Translation: We cut out the time on managing our schema and sped up our development, and this meant we got to spend it all back and more managing schemaless storage!
If you stop refactoring. So you know what? Don't stop.
At the beginning is the project when you know the least. About your tech, about your product, about your customers, about your competitors and partners. The best design decisions are made with the most information. Ergo, all design decisions should be delayed as long as possible.
The only question is how we responsibly delay design decisions. The answer for me is lots of automated tests and a willingness to refactor as we see ways to improve our designs. Plus a bunch of other agile technical practices.