You've done everything right: you have a continuous deployment pipeline, you have a commit stage with great unit tests, a pre-prod stage with fully automated deployment, robust end-to-end tests as well as some performance & security checks, and once all your quality gates are passed, your stream of changes march confidently towards production - but then somehow, someway, somewhy, production still blows up!
Continuous deployment is awesome, but the devil is in the details, and there's a decent chance you've been bitten by one of the biggest challenges - the safe management of contract changes.
Contracts can be obvious and public such as APIs (URL structure & schema), or obvious and private, such as database schemas, or non-obvious, such as the schema of in-flight messages in a queue, or dependency on specific timing expectations.
This talk demonstrates with real deployments the fundamental principle that underlies all safe contract changes - the 3 step model:
The good news is, with continuous deployment we can often do all three steps in quick succession!