Best practices for successful implementation of continuous delivery. Part 2

This is the second of three blog postings on the concept of continuous delivery. It’s an important but often overlooked process that can really help SaaS companies improve their efficiency. The first installment explained the concept and listed reasons why companies need to pay attention to it. This blog explains the eight core practices of successful continuous delivery. The final posting in the series will present simple tactics to let you get the most out of the process by focusing on team culture.

In the last post, I explained that continuous delivery is an efficient and effective way to care for your software by making product releases go more smoothly.

In this post, I will describe the eight core continuous delivery practices we use at Klipfolio:

1. Refocus your development process on continuous delivery
2. Be strict about version control
3. Define and model your deployment pipeline
4. Use continuous integration for your projects
5. Create and implement a quality strategy
6. Use configuration management tools
7. Implement proper logging and monitoring system
8. Develop a downtime strategy for your software

Refocus your development process on continuous delivery

All software development teams follow a process. But when teams start thinking about continuous delivery, they have to reimagine that process.

For efficient continuous delivery, it is important to clearly define each step along the way. It is also important to use an issue tracking tool like Jira or Youtrack throughout the process. And you want to ensure that once a piece of code goes through all these steps, it comes out ready for production.

Agile teams often come up with a definition of when something is done. When you adopt continuous delivery, it’s critical to emphasize that ‘done’ means the product is in production and used by customers.

The following is an example of what an end-to-end continuous delivery development process could look like. In this process, we track units of work we call "issues" throughout their lifecycle, and not just through implementation. This is important, because as I noted in the first post, continuous delivery affects the entire business, not just the development team.

Be strict about version control

Version control is the cornerstone of any software project, and version control systems have been around for years.

If a team is serious about continuous delivery, it has to put everything into version control - not just the source code of the software. And by everything, I mean the build scripts, the automated tests, the code that is used to build the servers, the configurations and the monitoring scripts. This way, you can track, control, and undo any change that occurs in the production system. You can also understand who made a change and why.

Good version control (we use Git) can eliminate hours of guesswork if the last known good state is available at a moment’s notice. In addition to typical source control software, there are also artifact repositories (we use Nexus) that you can use to track builds and binaries.

Define and model your deployment pipeline

The deployment pipeline is the assembly line for your software development process.

In an ideal world, all changes to production, including server and configurations, should go through the same pipeline. And you would be confident that any change sent through the pipeline could be deployed to production immediately.

As changes go through the steps in the pipeline, your confidence in those changes increases. While an ideal pipeline would be fully automated, you may have to have a few decision points and human involvement steps (exploratory testing, for example) built into your deployment pipeline.

Defining the model for your ideal pipeline helps you plan your next steps and projects for improving your continuous delivery infrastructure. It also lets you be clear about what you get or don’t get from the deployment pipeline.

Here’s an example of what a deployment pipeline model could look like.

The pipeline should not be an afterthought in a software project. It should be put in place right at the start.

Use continuous integration for your projects

If your deployment pipeline only ends up having one stage, that stage should be continuous integration. Continuous integration has been around for a long time, but surprisingly there are still teams that don’t use it.

Continuous integration systems have two goals.

First, they give continuous and fast feedback to software developers. This allows them to react quickly and change their code if a problem arises with something they’ve just done.

Second, they protect the master branch and other team members from a bad code change by identifying issues and preventing bad code changes from being merged into the master branch. This can be done via continuous integration and version control tools integrations.

As a result, good continuous integration plus a proper branching strategy (see Github flow as an example) can save the team many hours of work.

The key success factor with continuous integration is giving priority to keeping the builds green.

Create and implement a quality control strategy

Often, the steps in the continuous delivery pipeline are indications of the quality bar for your releases. For instance, you can choose to have performance, load and manual exploratory testing steps in your pipeline.

Additional steps increase your pipeline cycle time, but can increase the quality of your releases.

Two strategies that we’ve found essential and have saved the day for us many times are:

• Feature switches: Release the feature to production, but put it behind a feature switch so that only users for whom the switch has been turned on see the feature.
• Canary releases: Deploy the code changes to only a subset of the servers.

Both of the above help with cutting the risk by exposing the features to only a subset of users. We always do a Canary release to a set of servers that are used only by Klipfolio users.

Find the balance between quality and cycle time.

Remember that frequent releases don’t necessarily lower quality. In fact, more frequent releases should increase quality, as you will be releasing smaller packages that are by definition easier to control.

Use configuration management tools

Configuration management allows you to change the configuration of your servers on all your environments, including development, test, staging and production.

Configuration management saves time by allowing, for example, mass changes to configuration if they are needed. For instance, you can change the database URI that all your servers point to in one configuration push.

Automating the configuration process allows for consistency and eliminates human error.

Like everything else, configurations should be stored in version control.

Externalizing configurations from your build binaries and using configuration management tools to apply the right configuration for the right environment also allows you to use the same artifacts on all environments - test, performance, production. This increases the confidence in the produced artifacts as they go through the various stages of the pipeline.

In its most basic form, configuration management can be done by pulling down configurations from your version control system. However, as the number of servers increases, that approach is not sustainable. You may want to think about a better solution at that point. We use a tool called Ansible.

Implement a proper logging and monitoring system

Continuous delivery without proper logging and monitoring is like flying blind. You need to know the state of your production servers, as well as the errors that may be happening on the client side in browsers and apps.

This is crucial because prevention of errors is only half the battle. Releases inevitably go wrong, and you need to catch and fix the issues ASAP.

Logging and monitoring enable you to monitor the impact of the release on customers. This is not only helpful for catching and reacting to bugs and errors, but also to see if fixes are actually being used successfully, and whether they solve the issues that were observed. We use a number of tools in this area, including Kibana (ELK Stack), Grafana and NewRelic, plus an in-house bot that integrates with Slack, our messaging tool.

Develop a downtime strategy for the software

Will customers be using the software 24/7, or will there be periods when the software is not in use?

The answer to that question significantly changes your approach to continuous delivery. It will have an impact on your tooling, your app architecture and your server topology.

It’s not easy to keep a service up 24/7 and yet do daily deployments.

You have to figure out how to do so. There are two main options if you want to do deployments without downtime and during the business hours:

• In a rolling restart, servers are (usually) stateless and they are updated one by one and restarted. While one server is being restarted, traffic is redirected to the other servers.
• In a blue-green environment, there are two identical production environments. You update one while the other one is in use. After the update, you direct traffic to the environment that just got updated.

Avoid doing off-hours deployment as much as you can. While off-hours deployment may feel appealing, it will burn out your team and prevent you from moving your architecture in the right direction so that you can do daytime and therefore more frequent deployments.

Deployments should not be big bang events that come with all-nighters and weekend all-hands-on-deck events. They should be just yet another task during business hours.

Here’s a chart showing the practices we use to make our continuous developments successful:

In the final blog in this series, I’ll look at how to turbocharge your continuous delivery process.

Ali Pourshahid, PhD, is Director, Software Development, at Klipfolio. He can be reached at @ali_pourshahid