Making the case for Continuous Integration

by Uwe B. Meding

Continuous integration (CI) has been around for quite some time now. Originally, the idea was conceived as part of the extreme programming (XP) practices. CI involves making small changes to sofware, and then building and applying quality assurance processes. Defects do not only occur in the code but also appear in the naming conventions, documentation, how the software is designed, build scripts, he process of deploying the software to servers, and so on. Continuous integration forces the defects to emerge early, rather than waiting for software to be fully produced. If defects are caught in the later stages of the software development life cycle, the process will be more expensive. The cost of repair radically increases as soon as the bugs escape into production. Some estimates suggest it is 100 to 1000 times cheaper to capture defects early ^1 2. Effective use of a CI server, could make the difference between enjoying a holiday and working unplanned hours to heroically save the day. At its very core continuous integration is a set of principles that apply to the daily workflow of the development teams.

Continuous integration workflow

1. All source code must be kept in a version control system or source repository such as Git, Subversion, Perforce, Mercurial etc. These repositories can be self-hosted or, better yet, you can choose one of the many hosted implementations such as GitHub, SourceForge and many more.
2. In a CI life cycle, when a developer checks in his or her revised  code into the repository, an automated system picks up the change and checks out the code and runs a set of commands to verify that the changes are good and did not break anything.

The CI server and workflow is meant to be the unbiased judge of whether a change works or not, thereby preventing the works on my machine (WOMM) excuse before the code hits production. For all this to work, the build must be automated. A build ideally involves more than just compiling, it should also include a thorough testsuite to help verify that the code still works with every change. The objective is to increase test coverage with every new change. However, this needs to be orchestrated with care, because we also need to keep the testsuite fast. Pick a test suite software that lets you run tests incrementally, or an arbitrary order. Interactive test runs should take no more than a few mins, or developer producivity drops, slowing down the process of shipping new features or bug fixes to the customers.

Continuous Integration Overview

Continuous integration practices

Commit early, commit often

A core practice of CI is that all developers commit to the main line branch (aka trunk) daily. With modern distributed version control systems such as git the temptation is high to work on a local branch, maybe pushing your work to a remote repository and eventually merging your changes. Alternatively, what if all developers made all their change on the main line branch?

When your team makes changes in smaller increments and integrates them into the main line regularly, the benefits go beyond just the development process. Smaller changes shipped to production quickly are a lot easier to to debug when something breaks. This is one of the key values of CI and it is where the name comes from. Rather then living in branches for long chunks of time, changes are continuously integrated.

Build early, build often

How do you verify whether someones changes broke the code or whether the changes work in the larger context of the system of the entire code base. Beyond version control, a continuous integration server is one of the more important tools a development team can put to good use. Its sole purpose is to check the code repository for changes, check out the code if it spots any, and run a set of commands that trigger a build.

The definition of a build varies widely depending on the programming languages and frameworks used. With compiled languages such as Java, C++ and the like, a build is a compilation step with the additional run of a test suite. For dynamic languages such as Ruby, Python, and JavaScript, a test suite is commonly the only thing that  is run as part of the build process. A CI server is unbiased — its task boil down to telling the team whether the most recent changes still pass the stages that its configured to run.

The last step of a fully automated build, is the ability to deploy to production which reduces an automated deployment process that every developer should be able to run just like the CI server. With the automated build in place everyone can deploy to staging or production any time. Imagine shipping all new code directly to the staging system for instant testing.

Keep it fresh

With all developers committing their changes to the main line, a green build is an important responsibility. A green build is the result of a code change passing all the steps involved in the bulld successfully. When developers break the build, they are blocking everyone else on the team from commiting their changes with confidence. When the build is broken and other developers keep making changes and commiting them, its a lot harder to figure out whether they fail the build or not making debugging the failed build harder with every new commit. At Toyota, for example, everyone on the factory floor can halt the production line if something breaks or holds them up. In your development team, a broken build should spur a similar habit. Getting the build back to green should be highest priority. A green build is your insurance policy that changes can be deployed to production at any time.

Continuous integration culture

Beyond having good tooling and a fast test suite in place, CI is really about fostering responsibility and providing customer value. The server running the build process is but a small part of it. The practices that continuous integration includes have much bigger picture in mind. A responsible team insures new features can be shipped to production and the customer quickly and that makes sure all the team members can get their work done and check in new code. When new changes are continuously merged into the main line branch, there is a deeper responsibility for everyone on the team to make sure the code changes not only pass the build but that they also have minimal impact on the production environment. Rather than build new an bigger features on long lived feature branches, developers start thinking about ways that let them merge their work into the main line branch regularly. Feature toggles are a popular way of reducing the impact of new features and provide ample means to try them out before shipping to all customers. they have the added benefit that when something breaks in production, features can easily be switched off to reduce the exposure on the system while the issue is investigated. How do you notice when something is broken? How can you know that a change does not have a negative impact on the production environment or on your customers? Good monitoring is essential. Logging, metrics, alerting and exception tracking will help you ship changes with confidence. Thanks to an abundance of hosted services, its just as easy to get started with monitoring as it is with CI servers. Feature toggles and monitoring are not practices originally considered with CI, however, they are invaluable additions to the process of shipping early and often. When you add these habits together, along with feature toggles, fully automated and fast deploys, and monitoring you are ready to ship value to your customers any time

Continuous integration resources

An abundance of tools are available from self-hosted systems such as

• Jenkins
• TeamCity
• Bamboo

to hosted products like

• CloudBees
• Travis CI

They are all straightforward to set up, so there is no excuse not to use a CI server for your projects.