Post-Deployment Code Review

There may be times when rapid prototyping is more valuable than production stability — for example, in the early days of a service’s creation, or in the event of an operational issue, or if the service performs background tasks that aren’t customer-facing.

In situations like these, code review becomes a hindrance instead of a help. Indeed, there are times when “Does this change work in practice?” will be the first question on a reviewer’s mind. With enough effort, of course, anything can be tested in pre-production environments, but doing so in an early, experimental phase isn’t always practical or worthwhile.

But if we skip code review before build and deployment, when does the code actually get reviewed? Most git tools are designed for a pre-merge, pre-deployment review process; once a change is in master, it’s never looked at again.

In this article, I propose an addition to the standard trunk-based workflow using a new branch, the flux branch. Changes in the flux branch go out to production, but aren’t considered “final” until flux is merged to master. This allows for code review to take place during a pull request from flux to master.

During the development cycle, builds from both master and flux will go out with an identical change history — master+flux — to ensure the code running in production is always in a known, consistent state. Then, once the code has been accepted in review, flux is merged to master and standard trunk-based development resumes.

The flux branch workflow

The flux branch is a short-lived branch designed to supplement a trunk-based workflow.

Development starts by cutting the flux branch from master. As the developer(s) working on the flux branch make commits, they pass through automated linting and testing steps, but are not reviewed before deployment.

While the flux branch exists, all builds from flux and master should go out with an identical history, with flux automatically rebased onto master at build time. This way, no matter which branch build is triggered, the same code is deployed — even if master diverges from the original flux point during the course of development.

Once flux is ready for review, the developer(s) working on flux should rebase it against master for ease of comparison and merging.

After a successful review, flux should be merged (fast-forwarded) into master and then deleted.

Work can continue in the normal trunk-based fashion until such time as it is determined that a new flux point is needed, and then the process starts again.

Automatic rebasing in Jenkins

The flux branch workflow hinges on the build system performing automatic rebases on both master and flux builds.

Since I use Jenkins as my build system, there’s one extra consideration: by default, branch and project scanners won’t configure refspecs that include remote branches. You’ll want to configure the scanner to work with remote branches.

Here’s some example Jenkins library code to do the rebases on build.

def autoRebase() {

    // It's necessary to set up an identity for the CI server
    // for when master diverges. otherwise, git refuses to rebase.
    sh('git config --global user.email "ci@awk.space"')
    sh('git config --global user.name "ci"')

    if (BRANCH_NAME == 'master') {
        def branches = sh(
            script: 'git branch -a --format "%(refname:short)"',
            returnStdout: true
        ).trim().split("\n")
        if ('origin/flux' in branches) {
            sh('git rebase --empty=keep HEAD remotes/origin/flux')
        }
    } else if (BRANCH_NAME == 'flux') {
        sh('git rebase --empty=keep remotes/origin/master')
    }

    // Useful to verify that the recent commit history looks right.
    sh('git log -10 --oneline')

}

A side effect of this is that builds will fail if the rebases cannot be performed automatically. In those cases, flux should be manually rebased to master.

Closing thoughts

This development pattern has limited use under a narrow set of circumstances. It’s great for experimentation if not stability. So far, I’ve found it plays nicely with tools designed for the typical pre-merge workflow.