"After all, the engineers only needed to refuse to fix anything, and modern industry would grind to a halt." -Michael Lewis

Enable Massive Growth

A Simple Zero Downtime Continuous Integration Pipeline for Spring MVC

Nov 2018

The sample code associated with what follows can be found on GitHub.

One of the biggest paradigm shifts in software engineering, since the invention of the computer and software that would run on it, was the idea of a MVR (minimum viable release) or MVP (minimum viable product). With the lack of internet access becoming the exception in developed countries, it becomes more and more powerful to put your product out there on display, and to design a way to continuously make improvements to it. In the most aggressive of circumstances, you want to be able to push something up to a source control server, then let an automated process perform the various steps required to actually deploy it in the real world. In the best case, you can achieve all of this with zero downtime--basically, the users of your service are never inconvenienced by your decision to make a change. Setting up one very simple example of that is the subject of this post.

I'll start with a skeleton Spring MVC project. You can use the Spring Initializer with the Web option to get started quickly and easily. All I'll do, to keep the focus on DevOps, is add a simple entry controller that will accept requests to the root directory:

public class HelloWorldController {

    @GetMapping(value = {"/", ""})
    public ResponseEntity<String> notAnotherHelloWorld() {
        return new ResponseEntity<>("okay, I guess we'll call this a hello world", HttpStatus.OK);

As you can see, I'm reluctantly calling this a "hello world." Yuck.

And that's it for the Spring MVC project. What we need on the server is:

  1. Dependencies (e.g. java)
  2. A source control server (to checkout the code and perform the steps on commit)
  3. A way to build the solution in a way that is separate from running our solution
  4. A way for the web application to run continuously, and to restart automatically if it crashes
  5. A reverse proxy server to balance requests between two active applications. When one is stopped for upgrades, the second can continue to run, and vice versa.

I will start with #4: we can create a systemctl service (call this site-node1.service):

Description=Site Node 1

ExecStart=/usr/bin/java -jar -Xmx64m /opt/site-node1/target/simplecicd-0.0.1-SNAPSHOT.jar


This runs with the environment variable SERVER_PORT equal to 9000, which tells Spring to run our application on localhost:9000 (which will be the local host on the server). We can create another service file for another port (9001, here) like so (call this site-node2.service):

Description=Site Node 2

ExecStart=/usr/bin/java -jar -Xmx64m /opt/site-node2/target/simplecicd-0.0.1-SNAPSHOT.jar


By choosing to use the local host to bind these systemctl service files to, we are capable of using a reverse proxy. Typically, a lightweight server application gets the job done from here. I will choose Nginx, and create a dead simple configuration file like so:

upstream nodes {
        server localhost:9000;
        server localhost:9001;

server {
        # balance between nodes
        location / {
                proxy_pass http://nodes;

Here, we use the load balancing feature to go between the two nodes defined in our service files above. Nginx will automatically detect if a port is not in use, and will not forward to that port if there is nothing to forward it to. For example, if both nodes are running and we decide to kill the one running on localhost:9000, Nginx will route all requests to localhost:9001.

I will use Git for source control, and will use git's hooks feature, particularly the post-receive file, to run a bash script on check in. The bash script will build the solution (including running any tests) and, if successful, will copy the directory with the built solution into our predetermined folder, then restart each service. Before starting the second service it will validate that the first one came up successfully, thus never shutting down both services at the same time:


# checkout changes to release branch
git --work-tree=/opt/build --git-dir=/srv/git/site.git checkout -f release

# build the solution with maven and ensure it passes all the tests. If it fails, abort
cd /opt/build
mvn clean install
if [[ "$?" -ne 0 ]]; then
  echo "build failed, stopping deployment"
  exit 1

rm -r /opt/site-node1/*
cp -r /opt/build/target /opt/site-node1/
echo "restarting the first service"
systemctl start site-node1.service
systemctl enable site-node1.service

START_TIME="$(date +%s)"

while [[ `curl -s -o /dev/null -w "%{http_code}" localhost:9000` -ne 200 ]]; do
  sleep 2s
  if [[ `expr $(date +%s) - $START_TIME` -ge 60 ]]; then
    echo "timed out--something is wrong. Aborting..."
    exit 1

echo "first service up, restarting second service"
# stop the second service
systemctl stop site-node2.service
# copy the target directory from the build into the second node directory
rm -r /opt/site-node2/*
cp -r /opt/build/target /opt/site-node2/
# restart the second service
systemctl start site-node2.service
systemctl enable site-node2.service

To stitch all of this together, I will use an ansible playbook to provision our server, and I will use Vagrant as a proof of concept. Here's the playbook, which sets up the environment we need to make all of this work:

- hosts: all
  become: yes
  gather_facts: no
    - name: 'install python2 on ubuntu 18'
      raw: test -e /usr/bin/python || (apt-get -y update && apt-get install -y python-minimal)

    - name: Gathering facts

    - name: Get Java tarball
        url: https://download.java.net/java/GA/jdk10/10.0.1/fb4372174a714e6b8c52526dc134031e/10/openjdk-10.0.1_linux-x64_bin.tar.gz
        dest: /etc/open-jdk10.tar.gz

    - name: make java 10 directory
        path: /usr/lib/java10
        state: directory

    - name: unpack tarball
        dest: /usr/lib/java10/
        src: /etc/open-jdk10.tar.gz
        remote_src: yes

    - name: update alternatives for java
        name: java
        path: /usr/lib/java10/jdk-10.0.1/bin/java
        link: /usr/bin/java
        priority: 2000

    - name: set java environment variable
        insertafter: EOF
        path: /etc/environment
        block: export JAVA_HOME=/usr/lib/java10/jdk-10.0.1

    - name: re-source env variables
      shell: . /etc/environment

    - name: install packages
        name: '{{ item }}'
        state: present
        update_cache: yes
        - nginx
        - maven
        - git
        - python-psycopg2

    - name: allow ssh
        rule: allow
        name: OpenSSH

    - name: allow http/https
        rule: allow
        port: '{{ item }}'
        - 443
        - 80

    - name: setup rate limit over ssh
        rule: limit
        port: ssh
        proto: tcp

    - name:
        state: enabled

    - name: create git directory and node directories
        path: '{{ item }}'
        state: directory
        mode: 0777
        - /srv/git/site.git
        - /opt/site-node1
        - /opt/site-node2
        - /opt/build

    - name: create git repo to push to
      command: git init --bare /srv/git/site.git
        creates: /srv/git/site.git/HEAD

    - name: copy node service files
        src: '{{ item }}'
        dest: /etc/systemd/system
        mode: 0755
        - ./site-node1.service
        - ./site-node2.service

    - name: copy nginx config file
        src: ./nginx_site_conf
        dest: /etc/nginx/sites-available

    - name: make link to sites-enabled
        src: /etc/nginx/sites-available/nginx_site_conf
        dest: /etc/nginx/sites-enabled/nginx_site_conf
        state: link

    - name: remove default configuration
        path: /etc/nginx/sites-enabled/default
        state: absent
      register: nginxconf

    - name: reload nginx
      command: nginx -s reload
      when: nginxconf.changed

    - name: ensure nginx runs on boot
        name: nginx
        enabled: yes

    - name: create post-receive file to deploy solution on check in
        src: ./post-receive.sh
        dest: /srv/git/site.git/hooks/post-receive
        mode: 0777

If you go get the source code and install the prerequisites, you can validate that this works by running:

$ vagrant up

Then, once the VM is provisioned:

$ git remote add local_vagrant root@


$ git push local_vagrant release

And watch the magic happen.

Nick Fisher is a software engineer in the Pacific Northwest. He focuses on building highly scalable and maintainable backend systems.