configly

[configly]

Ways to reduce the pain of deploys

In our last post, we talked about the cost of deploying code. In this post, we are going to talk about ways to reduce the pain of deploys. There are a few different strategies to accomplish this. The first is to use tooling that makes deploys faster and easier. The second is to reduce the weight of deploys. And the third is to remove some things from the deploy cycle altogether. By combining these strategies, you can reduce the pain involved with deploying and speed up your release cycle.

Improving deployment tooling

There are a number of ways to reduce deployment costs by rolling out improved tooling. Platforms that enable continuous integration (CI) and continuous deployment (CD) can make deploys both easier and much faster. They can automatically run your tests or other verification processes, preventing bad code from being deployed in the first place. When the code is ready to go, it can be deployed automatically. This will also reduce the amount of code that is deployed at a time, allowing you to isolate bad commits and fix them faster. 

A more recent development is continuous verification (CV), which gives you tools that you can use to determine the quality of a new build when it is deployed in production. In some cases, you can deploy a canary build to assess performance and error rates in relation to the stable build. In the case when bad code is deployed, CV tools can let you know of issues that arise and even potentially roll back automatically.

Platforms such as CircleCI, Harness, Jenkins, and Spinnaker can all help you to enable CI/CD and even CV.

Reducing deployment weight

A second way to make deploys less costly is by decreasing deployment weight. This can be accomplished by adopting CI/CD, as it will be possible to deploy smaller updates more frequently. However, another way to do this is by breaking up your application into smaller services that can be verified and deployed independently. This will enable you to deploy smaller pieces of code with a smaller blast radius.

This is not to say that microservices are a panacea; the Internet is littered with horror stories of microservice migrations that took years and introduced a host of new problems. A lot of companies either don’t put enough work into setting up tooling for microservices, don’t think enough about standards or interoperability, or divide their app into way too many microservices. However, if you think intelligently about how to split up your app and do the migration deliberately and thoughtfully, the result will be a more robust application that is easier to deploy and test.

There are a number of platforms that can make it easier to split up your app into smaller services. These include Spring Boot, Akka, Kubernetes, Docker, Prometheus, and many more.

Removing data from the deploy process

The final strategy for making deploys less costly is to remove some aspects of your code from the deployment process. When they are starting out, developers often hardcode various things in code, including configuration constants, user-facing text, and even feature toggles. While this is often easiest in the short term and will work when deploys are fast, it becomes progressively more annoying to push code every time you want to change a piece of text. A lot of companies end up solving this with a homegrown solution, which typically involves storing the data in some type of database. This will work, but even then the data isn’t easily accessible to non-coders. You build a UI to allow non-coders to edit the data, but this will require additional work.

So how do you remove data from the deploy process? The first option is to use a generic config server. This will allow you to update text or constants on the fly, and changes will be reflected in the production application without a deploy. Options for doing something like this are Firebase Remote Config and Configly.

If you want a tool more suited to specific applications, you can find tools that are suited to almost any common use case. They will require some upfront work to implement, but will pay for themselves in the longer-term. If you want to internationalize your text and remove it from your code, you can use Simplelocalize, Lokalise, or Locize. For feature flagging, consider Launch Darkly, Split, or Apptimize.

These are just some of the many ways that you can make your deployments lighter weight and faster. By making deployments easier, you will both reduce frustration and make it easier for your team to move faster.

[configly]

The Cost of Pushing Code

When you begin building a new product, pushing code to production is mostly effortless. For a server or web application, you just push the code to the main branch and run a deploy script. In some cases it is possible to set up a “Git push” deploy (e.g. Heroku), which automatically deploys whenever a branch is pushed. This workflow makes it super easy to keep everything in code, including your configuration and all of your product copy. However, as your product and company grow, it becomes harder and harder to push new code. This post will go over the major reasons why pushing code becomes painful, and discusses alternatives that may reduce some of this pain.

Time and Effort

The first pain point is that it takes time and effort to deploy code. A lot of companies don’t do continuous or even daily deployment, and code changes often go out between a day and 2 weeks after they are committed. This introduces a significant amount of lag, and prevents you from moving quickly when something minor needs to change. It might be possible to override the process and push a hotfix to production, but there is a time and effort cost to this as well.

Once you get the green light to deploy, there is still a significant amount of work to actually get the code out. Even though the push process may appear simple on the surface, most companies have pipelines that build the code and run at least some unit and integration tests. Every time you want to deploy something new, you need to wait for these tests to run, and if anything breaks (potentially from an unrelated commit that someone else pushed to the main branch), you need to track down the problem before you can deploy. This adds mental overhead, and can discourage you from even trying to make a change in the first place.

Furthermore, a lot of companies have both production and staging environments. Before you can deploy to production, you must deploy and verify your changes on staging. These environments can be total lifesavers when used appropriately, but unfortunately they do add time and cost to deploying code.

Finally, it bears mentioning that every code change requires engineering time. Engineering time is expensive, and engineers don’t respond well to distractions. Even a 1-line copy change may cost you and hour of engineering time that could be spent on other higher-value activities.

Unexpected Consequences

The second major pain point is that pushing code can have unexpected consequences. Even something as simple as changing a string in text can have wide-ranging consequences if that string is used in numerous places. Another example would be a configuration change. If someone makes a simple config change that invalidates a piece of JSON or even just changes its format, the entire application could fail to start. There have been plenty of times in my career where a seemingly innocuous change took down a whole app. While defensive programming practices can reduce the effect of an erroneous configuration, these sorts of things are frankly unavoidable.

Even the fear of breaking things can make it more difficult to change code. Every developer has some level of fear that he will accidentally push a bad commit and then get fired (or publicly humiliated) over it, and this leads most engineers to become somewhat risk averse. Accordingly, there have been many times where I didn’t deploy a “simple” change immediately because I was afraid that it might cause something bad to happen. In the best case, this type of fear will delay these changes until the next morning or to spend hours monitoring metrics to verify that nothing is broken. In the worst case, it could cause people to decide “not to bother” in the first place.

Gatekeepers

The third pain point is that there are often numerous gatekeepers who must be placated when pushing code. First of all, only engineers can make code changes. This means that if all of your product copy is stored in code, and a product manager or marketer wants to change that copy, then they must enlist the help of an engineer before they can make the change. Likewise with configuration changes that are stored in code. If someone on the customer support team wants to override some limit that is hardcoded, then the only solution is to have an engineer push code. 

Then there are further gatekeepers, including the code reviewer and potentially devops (if they are the ones who run the deploy). Every code change will have to go through each gatekeeper, and even if the review is a formality, it is possible for a change to get stuck indefinitely due to a missed email or competing priorities.

Another gatekeeper for mobile applications is the app stores. Google and Apple review every app submission to their platform, and while the review process has sped up significantly in recent years, updates often still take 24 to 48 hours to review, and Apple occasionally rejects small updates for unrelated reasons. The result is that it often isn’t possible to “quickly” push a change when someone finds a typo, and it has the potential to become a long and drawn out process.

The Solution

So the solution doesn’t involve changing the code push process, which is there for good reason. As code deployment becomes painful, you will want to pull anything that isn’t code from your code base, and put it into systems that you can quickly update. Configurations and copy can be moved into alternative systems, and pulled by the server or client whenever needed. For example, you could put them in an S3 bucket or any other key/value store. While this takes a little while to set up, It frees you from the tyranny of pushing code whenever you need to make a small change. Over the long term, this sort of system will save you far more time and money than it costs. Finally, I would be remiss if I didn’t mention Config.ly, which makes it super simple to edit all kinds of string and JSON data, and then to fetch them in real-time from your server or directly from your mobile applications.