When I’ve talked with developers about CI/CD the main problem is always:
“I don’t know how to work in this way”
I will try in this article to explain how I do it.
From feature to commit
The most important thing in my opinion CI/CD brings us is switching the perspective from feature to commit. This means that you are not deploying features you are deploying commits, in terms of Lean Agile our batch size now is the commit. So we have to be very sure our commit is working and is not breaking anything:
- Each commit needs to be tested, no commits without tests. All the tests must pass.
Ok, but we still can push all our feature in one commit. We need to deploy frequently to our integration and production environments, so I have the rule of the time:
- I have to commit in less than x minutes (you can start by 30 and go down to the number you want, even seconds). If you cannot commit in that time reset to your last working commit and start again, probably you will have in mind a better plan to do it faster.
This second rule is interesting because if your time box is little you will accept easier to remove the code but if you last too much time then this is not going to happen. As much time we are in a broken state in our build more bugs we will introduce (more things we will break).
We want to minimize the time we have our code broken, if our code is healthy then the possibilities to introduce bugs are reduced a lot.
Yeah, but what if I have to change legacy code, how can I spend 10 minutes to commit with passing tests?.
When I have to add a new feature to one system I usually start playing with the code trying to understand where is the point to introduce the code for the new feature. During this time I can do whatever I want, it is not needed to make passing the tests, everything can be broken, but I cannot commit anything. The intention of this time is to learn from the code and to have a little plan of what to do (write your plan somewhere, not just in your memory). Remember you cannot commit anything during that time you only have to create a mini-plan of what you are going to do and where you will do. The mini-plan created must follow the rules explained before and don’t need to solve the whole problem just a few steps to start working in the direction of creating the feature.
Branch by abstraction
Once we know where we will introduce our changes, we will follow one of the SOLID principles. Open/Close means in general that it is better to create something new than changing something old.
Let’s imagine we have to change completely a part of the code by another. We want to create a seam (a seam is a place where you can alter behavior in your program without editing in that place), to create the seam our plan could be something like:
- Extract the code we want to change to a method inside the class.
- Move that method to another class. This will create a collaborator in our main class.
- Extract an interface of our new collaborator and use that interface in our main class.
- Put the interface as a parameter of the constructor of our main class. In the second step we have introduced in our constructor the creation of the collaborator, so we can now extract as a parameter. In this way all the clients that use our class will create our new collaborator.
Each one of the steps described above can be done through refactors available in our IDE (Eclipse or IntelliJ for example), they can be committed and pushed once the step is finished because all tests if they exist are going to work.
Now we have a seam, we have the ability to introduce a new implementation of our class without changing the old, this is branch by abstraction.
So we can create our new implementation:
- Create a new test for each old test that you have in your main class using your new collaborator (if you follow TDD this will produce the creation of the new collaborator). Don’t remove the old tests.
- If you don’t have tests in the main class then just create new tests for your new implementation (your feature).
- Change in all your clients the creation of the old seam Class by the new one.
- Remove the old implementation and the old tests, they are not used in production.
- Remove the interface and use your new collaborator.
- Rename your new collaborator to a better name (probably the one selected at the beginning is very bad).
Again all these steps can be committed and pushed in that order, everything is going to work after each step is finished.
Now you have your new implementation working in production and each step you did maintain green all your tests. This is the great part of working in trunk based development you have to improve your skills to work securely all the time, and you write your tests during your code (TDD) or some minutes after, you receive early feedback about your design.
If you want to know more techniques to add code securely this is your book Working Effectively with Legacy Code.
Sometimes a change cannot be done only in one deliverable, sometimes you have to deploy different parts of your system. When you work in feature branching model you coordinate your deployments and if something is wrong this usually means to rollback everything.
In trunk based development this is forbidden in someway you cannot coordinate anything because everything is automatic. To make all this happens we can decouple the deployment of one feature from the activation of that feature, this is called feature toggles.
In the above example we could introduce a feature toggle. When we finished the creation of our new feature we could introduce a factory to create the old implementation or the new implementation based on a value stored in any database (an if). In this way we can control when to activate or deactivate our feature just setting that value to true or false in real time. With this approach we could have that value set to false until the whole feature in all our services have been finished, and we think our integration is working (we could enable that toggle in our integration environment and see how the integration is working).
So we have a great way to reduce the impact of a bug in production, if we activate our new feature, and we realize that we have a bug then we can disable it at that moment. No need to rollback all the services, no need to wait for anything the team is able to enable/disable a feature when they want. MTTR (mean time to restore) is reduced to the minimum possible value.
If everything is working as expected, then we have to create a new task to remove our feature toggle to enable by default our new branch, this usually means to remove the old code, the factory and use directly our new class. This step is very important because feature toggles mean complexity for your code (technical debt), so they have to be in your code the minimum possible time.
Don’t break the build
CI/CD also means to have a pipeline to deploy your changes to your environments (staging and production for example). Once you push a commit then your CI pipeline must build your code, execute your tests and deploy your code in one of your environments in each step. If you have the build broken because one of your tests is not passing or your code cannot be compiled then you have to stop doing what you were doing and focus on fixing your problem. If your build is broken all your team is stopped, so it is very important to solve the problem as soon as you can, if this is going to take you a while revert commit by commit until the build works again and fix the problem locally in your computer. In git, you can use git-bisect that will say you the broken commit quickly. The main important thing is reduced the time when you broke the build to avoid stopping your teammates.
In one team I was part we decided to sound a nuclear bomb when the pipeline was broken. After some weeks the sound was not needed anymore, all the developers understood the idea.
But if I cannot coordinate deployments how can I change completely an API from a service?
- We can create a new API in our service that is not called by our clients yet. Deploy our new API and client by client change our code to use our new API. After all clients have been changed we can remove the old API controller.
What if we need to change a field of an API that is being used?
- We can add a new optional field to the controller that receives the calls of the API and use that value if exists or the old one if the new is not there (use the bridge pattern). Later we can change client by client to pass the new value and the old one, once all clients have been changed we can ignore in our service the old value and use only the new one. And remove later client by client the old value.
These are only examples but as you can see it is just a question of thinking. The word here is backward-compatible.
Is Code Review a problem for doing CI/CD?
My answer is not, the problem are pre-merge code reviews. If you commit and push directly to master to go quick to production it has no sense to have a manual step in the middle that blocks until one reviewer is finding time to take a look to your code.
But how can we achieve the quality that code reviews gives to our code?
- Pair Programming: One friend of mine says that Pair Programming is a continuous code review. No waiting time, all discussions happen while the code is written.
- Post-merge code reviews: It is possible to do code reviews to commits merged to master.