"After all, the engineers only needed to refuse to fix anything, and modern industry would grind to a halt." -Michael Lewis

Enable Massive Growth

How to Provision a Consul Client-Server Cluster using Ansible

Apr 2019

The source code for this blog post can be found on GitHub.

Consul can run in either client or server mode. As far as Consul is concerned, the primary difference between client and server mode are that Consul Servers participate in the consensus quorum, store cluster state, and handle queries. Consul Agents are often deployed to act as middle-men between the services and the Consul Servers, which need to be highly available by design.

Ignoring my opinion about the architecture choices, we will expand on the last post (How to Provision a Standalone Consul Server with Ansible) and modify our ansible role to allow for agents to join the standalone consul server.

Because the default Restart=Always behavior of systemd isn't automatically honored, and we will need for the Consul Agents to restart while they try to connect to the server (which could still be coming up), the first thing we will do is get our Consul systemd service to keep trying to restart ad infinitum. Modify our templates/consul.service.j2 file to look like:

Description=solo consul server example

WorkingDirectory={{ consul_config_dir }}
ExecStart={{ consul_install_dir }}/consul agent -config-dir={{ consul_config_dir }}


Because the Consul Client and Consul Server instances will be on different virtual machines, we will need to add one for the Consul Client in our molecule/default/molecule.yml file:

  name: galaxy
  name: vagrant
    name: virtualbox
  name: yamllint
  - name: consulServer
    box: ubuntu/xenial64
    memory: 2048
    - "customize ['modifyvm', :id, '--uartmode1', 'disconnected']"
    - auto_config: true
      network_name: private_network
      type: static
  - name: consulClient
    box: ubuntu/xenial64
    memory: 2048
    - "customize ['modifyvm', :id, '--uartmode1', 'disconnected']"
    - auto_config: true
      network_name: private_network
      type: static
  name: ansible
        is_server: "false"
        node_name: client
        is_server: "true"
        node_name: server
    name: ansible-lint
  name: default
  name: testinfra
    name: flake8

The two main things that we have done here are:

  • Added a virtual machine, called consulClient
  • Set two host variables for both the consulClient and consulServer virtual machines. We will use them in our next step.

As luck would have it, we do not need to make any changes to the tasks/main.yml file. The only thing left to make this playbook "just work" is to modify the templates/consul.config.j2 file to look like:

    "node_name": "{{ node_name }}",
    "addresses": {
        "http": "{{ ansible_facts['all_ipv4_addresses'] | last }}"
    "server": {{ is_server }},
    "advertise_addr": "{{ ansible_facts['all_ipv4_addresses'] | last }}",
    "client_addr": " {{ ansible_facts['all_ipv4_addresses'] | last }}",
    "connect": {
        "enabled": true
    "data_dir": "{{ consul_data_dir }}",
{% if is_server == 'false' %}
    "start_join": [ "{{ hostvars['consulServer']['ansible_all_ipv4_addresses'] | last }}"]
{% else %}
    "bootstrap": true
{% endif %}

If we are running in server mode, we need the up and coming standalone server to bootstrap itself, hence "bootstrap": true. If, instead, we are running in client mode, we need the client to find our server to register with. Since molecule automatically adds inventory based on what is defined in the platforms section, we can reference our consul server's IP address in start_join.

If you run:

$ molecule create && molecule converge

You should be able to hit and see both the consul server and the consul client connected.

Be sure to go get the source code so you can play around with this yourself.

Nick Fisher is a software engineer in the Pacific Northwest. He focuses on building highly scalable and maintainable backend systems.