Continuous Delivery in a NutShell
Mainline Workflow
“Once you achieve a certain frequency of releases, around once a week or so, it no longer makes sense to branch for release.” — Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
tl;dr
I want to do a tl;dr of a workflow, that two years ago seemed insane, but now would seem impossible for me to work without and still feel efficient. The topic of Continuous Delivery and Mainline workflow.
To sum it up:
“Developers that practice working in small blocks of code, writing tests and committing often. Your app has the right amount of tests in place, a CI that runs with each commit, building and testing for every push. Reviewing each others code daily, and the ability to Feature Flag items that are not ready yet that with each push to the only branch, Mainline, the code then goes to production. [1]” Insane right?
But then when your hear of the many companies following this pattern:
- Amazon
- Atlassian
- WordPress
- Intercom
- IMVU, since 2009 (50 deployments/day)
- Wealthfront, since 2010
- Etsy, since 2010
- DISQUS, since 2011
- Box, since 2013
- Hubspot, since 2013 (300 deployments/day)
- Tesla and Adobe
[2]
And watch videos by Jez Humble and read the book Continuous Delivery, you start to see how you can leave behind a Feature Branch/PR workflow for this much more efficient and still stable workflow.
That about sums it up but now a bit more details.
Day to day responsibilities of the developers
Your team prioritizes keeping the software deployable over working on new features — Martin Fowler ContinuousDelivery
First of all this workflow really asks for developers to break down their work to small small steps/blocks. Almost going hand in hand with TDD.
With this, and some items I list below, the developer can move forward with confidence that the app is not going to break with the feature they are working on being not 100% done.
And the developer is working knowing that each block of code has to be production ready so there has to be trust here that they will code at this level. But all of this will surface in the code review that I will talk about below.
And to top it off they are pushing to the repo and therefore a Continuous Integration system like TravisCI, 3–4 times at least a day.
Testing
Being proud of 100% test coverage is like being proud of reading every word in the newspaper. Some are more important than others. — Kent beck
As noted above coverage and testing is key. This does not mean 100% coverage but as a team you have an agreement to the right amount and your CI system should fail when the code falls below this amount.
These test should be part of all our blocks of code you are committing and not be an after thought.
And lastly these tests are what give you all the confidence to have this workflow where you are pushing to a server numerous times a day.
Code Review
10 lines of code = 10 issues. 500 lines of code = “looks fine.” Code reviews — @iamdevloper
Code Review is key, and this is were many from a PR (Pull Request) work flow would argue is missing. Code is getting into “master” before being reviewed.
I get it, but if we are committing often, reviewing often and have good test coverage, then you will see areas of the code where patterns can be optimized or direction changed as you take a couple of times a day to do the code review.
And this is an pattern that will ideally help every one on the team to grow to the level expected. If some developers are truly too junior to code at this level then maybe there are more root level efforts to be had to help them, e.g. pair coding etc.
And the Code Review goes both ways not just lead reviewing others but reviewing each others code.
Feature Flags
Feature Toggles are a powerful technique, allowing teams to modify system behavior without changing code. — Martin Fowler
This is another key pattern to make this possible. Let’s say feature Foo is ready for production but feature Bar is not. Both are on mainline and both are pushed to production! But Bar is behind a feature flag/toggle. This means that unless an admin goes in and turns it on in the UI or a setting changed in the ENVIRONMENT is set to on then it will not appear, e.g. the load balancer sends traffic from us-east to server 1 where this feature is on.
As noted above the developer is working in small blocks that will not impact the application as a whole. Even migrations can be done this way. 2
This also leaves us open to do A B testing too if we want. Or better yet remove a feature that we find out was not being used! Nothing beats being able to remove code that is not needed.
CI
From now on, our code in revision control will always build successfully and pass its tests — James Shore: Continuous Integration on a Dollar a Day
This is the gateway from Developer to Deployment. This is the guard that keeps those accidental typos, missing libraries etc from making it to the server.
Just one .yml file later, good example here for php, and you are ready to use a system like TravisCI.
And this one step will be a world of difference even if you are working alone.
This is key in CD is that no code will be deployed till it passes CI. This means:
- The code will be built on the server catching “it works on my machine issues”
- The tests has to pass.
- The style rules have to pass
And once all this passes then you can have it do the deployment for you!
Even with tools like Dusk or Behat you can be testing your UI as well.
Deployment
Over time, deployments should tend towards being fully automated. There should be two tasks for a human being to perform to deploy software into a development, test, or production environment: to pick the version and environment and to press the “deploy” button — Humble, Jez. Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation
There are many ways to deploy of course. The CD goal is that you are deploy an “Artifact”. The reason is that this build and all the steps to make it so, that passed on your CI system should not be run again when deployed.
A good example of this is say you have 3 servers behind a load balancer and you want to deploy to all of them, one at a time. You then have to hope, if you are not deploying an artifact, that your steps will work on all them as they did on the CI. For me this would mean.
- composer install — which can have moments where Github fails or http issue
- yarn install — still some risk here of server issues
What, for me, has been working well is CodeDeploy. This integrates with TravisCI so it fits right into the .yml file. And during deployment of this artifact its is smart enough to do it one server at a time, removing it from the ELB (Elastic Load Balancer) while this is happening, and then putting it back when done if it passes. Stopping along the way and rolling back if there is a fail.
Long QA Branches
There are those moments when QA and regulations are blocking you. But this then turns into a “release branch” and not a tag.
Here are some quotes from the book Continuous Delivery.
“A more manageable branching strategy (when you can not release as often) — our strong recommendation, and arguably the industry standard — is to create long-lived branches only on release, as shown in the image above” from “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Adobe Reader) (Addison-Wesley Signature Series (Fowler))” by Jez Humble, David Farley
And
“Branch for Release The one situation when it’s always acceptable to create a branch is shortly before a release. Once the branch is created, testing and validation of the release is done from code on the branch, while new development is performed on mainline.” from “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Adobe Reader) (Addison-Wesley Signature Series (Fowler))” by Jez Humble, David Farley
Finally
“It is important, when branching for release, not to create further branches off the release branch.” from “Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation (Adobe Reader) (Addison-Wesley Signature Series (Fowler))” by Jez Humble, David Farley
Again this if for those times when there are systems / departments and regulations in place that leave you no real option.
Links
Jez Humble and Continuous Deliver 20017 Keynote: Continous Delivery Sounds Great By Jez Humble @ Agile India 2017 — YouTube
The Book here
Martin Fowler Continuous Delivery ContinuousDelivery
Feature Toggles — Feature Toggles (aka Feature Flags)
Footnotes
[1] Or even going to staging with an “artifact” will be good enough since that will be sent as a whole to production. ↩︎
[2] Credit to: What companies best exemplify IMVU-style continuous deployment? — Quora
[3] For us we keep logic in the code and not so much in the database, so when a field is created we tend to allow it to be null since this means we can manage that impact it has on the rest of the application if our code is not ready yet to manage the field. ↩︎
