CI/CD (Continuous Integration/Continuous Delivery) Best Practices and Trends

So, let’s explore some best practices for you to improve your CI/CD practice.

In a Nutshell: Continuous Integration, Continuous Deployment and Continuous Delivery

Large software teams run into problems when one person’s work messes with another’s without either of them realizing it. A clunky development process complicates the mission-critical stages of code integration and testing.

Continuous Integration discovers issues stemming from one person’s work interacting with another’s. Continuous Delivery ensures every change (configuration update, new feature or bug fix) committed to the code is releasable. Continuous deployment automates releasing the code to production. The CD pipeline incorporates performance, integration and security testing for code reliability.

A Day in the Life of the CI/CD Pipeline

After downing their first coffee, the developer pulls the latest code version from Git, using Tower, most likely. They develop new features or fix bugs. Then, they commit their work with the central repository for their team to access the updated version. This sets off the pipeline to detect the newest commit and trigger the build.

The code is compiled into an executable and packaged into a deployable. Next, the pipeline fires up the test stage. It deploys the code to the test environment and runs automated tests. If the tests pass, the code is deployed to the production environment. The pipeline communicates with the developer by sending build results to a Slack channel, creating tickets for failed builds in Jira and updating the dashboard with the pipeline’s latest performance metrics.

CI/CD Best Practices

1. Commit Early and Often With Minimal Branching

Committing code early and often eliminates the complexity and cost of merging huge and conflicting changes before a critical release. It’s best to use trunk-based development, where everyone commits changes to the centralized master branch to be automatically deployed to production.

While the trunk-based approach works best, code branches can help in the right context. For instance, branching can allow developers to work in parallel, isolate changes, and manage risk.

Standard branching techniques include feature branches (individual features are separate from the codebase), release branches (changes for a specific release branch out) or hotfix branches (code where a bug needs fixing branches out). That said, you should only use branches for a specific and limited-time purpose.

2. Place Security at the Front and Center

The CI/CD pipeline is fast, automated and vulnerable to security risks, like exposure of sensitive data, use of unsecured third-party components, and unauthorized access. Remember the infamous Uber data breach from October 2016, where hackers stole the personal information of 57 million users and 600,000 Uber drivers? Hackers discovered developers had published code with their usernames and passwords on a private Github, lending immediate access to the developers’ privileged accounts, along with the servers hosted on AWS.

Securing the CI/CD pipeline architecture involves using secure repositories for source control, conducting security checks during builds, running automated security tests and ensuring secure deployment practices. Helping developers to do this, we have DeployBot, that can assure that our code is reaching our servers in an automated and secure manner.

The next concern is to build security in the software. Integrating security tools throughout the pipeline- also known as DevSecOps- makes the software functional and secure. In practice, this looks like:

• Input validation
• Error handling
• Following principles of least privilege
• Performing static and dynamic code analysis
• Penetration testing
• Encrypting sensitive information

Remember, you don’t want to cut corners with code and pipeline security.

3. Only Build Once

Discard any practices where source code is built more than once. Even if you need to build, package or bundle the software before distributing, you should compile the source code once and promote your binaries. Creating a new build for each stage risks introducing inconsistencies.

Successful CI implementations start with the build process in a clean environment, which reduces the chances of oversight and error. You should then version and upload the artifacts repository to Git so that you get the original build whenever you pull it.

A prerequisite to implementing this principle is a system-agnostic build. So, you need to ascertain that any authentication parameters, variables, configuration scripts or files are called by the deployment script instead of being incorporated into the build itself. This way, the build is deployed to each setting for testing, and the team’s confidence in the artifact increases with each stage.

4. Know What to Test, When and Where

There’s the developer’s psychology- the more bugs in a program, the more wary developers are of fixing them- the Broken Windows syndrome, where visible signs of crime and civil disorder (broken windows) encourage further anti-social behavior. So, you’ve got to clean up regularly! If test results take too long, developers will look to bypass the process. Strike a balance between test coverage and performance since the purpose of CI/CD is quality at speed.

If you don’t use isolated testing environments, test locally before merging the code with the shared repository. The first set of automated tests are unit tests, which provide a broad coverage and highlight any obvious issues. Then, automated integration or component tests check interactions between parts of the code. Beyond these, you can perform some GUI tests, performance and load tests, security tests and acceptance tests.

Automated end-to-end functional tests are critical after unit tests. This image by the CTO of ibook.com with the caption “Unit tests all green, no integration tests” sums it up.

5. Have Only One Way to Deploy to Production

It’s how my gym trainer taught me to respond to my mind when it says, “Let’s skip it today” with “Sorry, no other way to start the day”. Even though skipping a day is possible, I’ve convinced myself it isn’t because going to the gym first thing is the best way for me to start the day.

Similarly, urgent and/or minor changes to the code can bypass the CI/CD process, but that’s a slippery slope. Circumventing the process can lead to bigger problems and nullify the benefits of CI/CD automation. Each change must strictly adhere to the established procedure and move through the CI/CD pipeline to safeguard critical environments from untrustworthy code.

Discipline is key to CI/CD success. So, it’s important that every team member sees the process as a reliable and efficient way to develop and deploy code. Also, by having the proper mechanisms in place, it shouldn't be a pain, but quite the opposite: an easy way to push changes into our servers, with the help for example of DeployBot.

6. Shift Left and Continuous Everything

Once you master CI/CD best practices, new ones emerge as the software development community is forever evolving. Having automated CI and CD pipelines positions you excellently for shift left. This means you move from issue detection to prevention by testing earlier and earlier to preserve code quality. The result is better software delivered quickly with a shorter feedback loop between developers and end users.

Shifting left isn’t just about the tools but about how everyone has a stake in delivering high-quality, reliable software. Product managers listen in on inquiry calls with the analyst relations team. Passionate customers test a feature and provide feedback to be inculcated in the product. No more islands of teams and BUs but continuous iteration and integration of everything- that’s CI/CD.