These kinds of rule-based migration safety checks are simple, but hardly complete.
The problem is that some migration safety depends on the state of the database, which isn’t represented in the DDL statement alone. For example, altering a column type is either a no-op or an exclusive locked table rewrite depending on the original type of the column.
There are other footguns that can happen if the column you’re altering is a foreign key, where multiple tables can be locked.
I went down a rabbit hole a few years ago and built a system[1] to introspect a given migration against a live schema, and actually let Postgres tell you what it’s doing[2].
It would be great to have better built-in support for this (EXPLAIN for DDL statements?), but this direction feels safer and more accurate than static rulesets.
Safety also depends on the size/activity of a table being altered (i.e rewriting an empty table is fine). Having an accurate representation of the locks and actions performed by the database lets you integrate with production metrics to actually determine real-world safety across a fleet of databases, rather than guessing.
Altering a column that already has data in production should be an absolute last resort, I don't think I've ever even done it, it's never 100% necessary
exactly: already has data. It’s not the statement that’s unsafe, it’s the size of the table. That’s what all pattern matching migration checkers get wrong.
You might be releasing a new feature gradually and you realised your schema is slightly wrong and want to alter a column type. You’ve got some tiny volume of data in one production cluster. Is it safe?
A pseudo rule determining the safety for any arbitrary migration that causes a rewrite could be:
smt.is_rewrite and tbl.size < 10MB
Yes: on your tiny new table
No: on your 10TB orders table
To accurately model migration safety you don’t really care about the statement: you care about the effects (locks, rewrites, additions, etc). That’s what is safe or unsafe.
Right I'm agreeing with you but if there even is a column that's already in production you should just assume it has data in it so I would favor just a blanket ban on altering columns at all
This largely depends on the kind of software system you are working on. These sorts of DDL migrations are common on the kinds of Rails apps I’ve worked on over my career. Size of the table, traffic patterns, who uses associated features, tolerance for small downtime windows, or orchestrating a multi-phase zero-downtime migration are all ways to justify these migrations, and is preferable to alternatives that would be comparably over-engineered for that app’s business and technical context.
orf · · focus · HN ↗
The problem is that some migration safety depends on the state of the database, which isn’t represented in the DDL statement alone. For example, altering a column type is either a no-op or an exclusive locked table rewrite depending on the original type of the column.
There are other footguns that can happen if the column you’re altering is a foreign key, where multiple tables can be locked.
I went down a rabbit hole a few years ago and built a system[1] to introspect a given migration against a live schema, and actually let Postgres tell you what it’s doing[2].
It would be great to have better built-in support for this (EXPLAIN for DDL statements?), but this direction feels safer and more accurate than static rulesets.
Safety also depends on the size/activity of a table being altered (i.e rewriting an empty table is fine). Having an accurate representation of the locks and actions performed by the database lets you integrate with production metrics to actually determine real-world safety across a fleet of databases, rather than guessing.
1. <a href="https://github.com/orf/locksmith" rel="nofollow">https://github.com/orf/locksmith
2. <a href="https://github.com/orf/locksmith/blob/f8798c6ee92bfae10d416c49aae33bc4afaaabf1/crates/locksmith/src/oracle.rs#L34" rel="nofollow">https://github.com/orf/locksmith/blob/f8798c6ee92bfae10d416c...
weird-eye-issue · · focus · HN ↗
orf · · focus · HN ↗
exactly: already has data. It’s not the statement that’s unsafe, it’s the size of the table. That’s what all pattern matching migration checkers get wrong.
You might be releasing a new feature gradually and you realised your schema is slightly wrong and want to alter a column type. You’ve got some tiny volume of data in one production cluster. Is it safe?
A pseudo rule determining the safety for any arbitrary migration that causes a rewrite could be:
Yes: on your tiny new tableNo: on your 10TB orders table
To accurately model migration safety you don’t really care about the statement: you care about the effects (locks, rewrites, additions, etc). That’s what is safe or unsafe.
weird-eye-issue · · focus · HN ↗
jbranchaud · · focus · HN ↗
weird-eye-issue · · focus · HN ↗