How to Configure Prometheus to Scrape and Aggregate Data From a Spring Boot 2.x Application

You can see the source code for this post on Github.

Following up on the last post [How to Expose Meaningful Prometheus Metrics In a Spring Boot 2.x Application], if we have metrics exposed but they don't go anywhere, are there metrics exposed at all?

Yes, there are metrics exposed, they're just not very useful. What we really want is to aggregate them and ship them to a data store so that we can view their evolution over time.

I'm going to use docker compose as a simple way to illustrate what you would need to do here. While you can run docker compose in a production environment, it has clearly lost (at this point - mid 2020) to kubernetes. I would recommend you either learn kubernetes or use virtual machines if you are planning on moving this to production.

Get the app running

We'll have to get the application running in our little docker compose network first. So we need to set up a Dockerfile and throw it in the root directory:

FROM maven

RUN mkdir -p /app

COPY ./ /app/

ENTRYPOINT cd /app && mvn spring-boot:run

At this point, we can set up the docker-compose.yml, which will represent a production like environment on our local machine:

version: '3'
  # `docker-compose build` to rebuild
  app:
    build:
      context: ../
    volumes:
      - /home/nick/.m2:/root/.m2
    environment:
      - SPRING_DATASOURCE_URL=jdbc:postgresql://db:5432/local
    depends_on:
      - db
    ports:
      - 9000:8080

  db:
    image: postgres
    ports:
      - 5432:5432
    env_file:
      - database.env

We are leveraging the isolated network that docker compose creates for us in this process and we are also leveraging spring configuration override. SPRING_DATASOURCE_URL is the same as if we would have specified a property in our application.yml that had something like:

spring.datasource.url: jdbc:postgresql://db:5432/local

When we tell spring boot to look up the datasource with the name db, we are deferring to that docker compose network, where db resolves to the ip address where the postgres container is set up. This will by default build the application as determined by the Dockerfile, then start it up. You should be able to run:

$ docker-compose up -d
$ curl localhost:9000/actuator/prometheus

And see some prometheus metrics output.

To wire up prometheus, we will want to use a built in prometheus image available on docker hub. We can add that to the services section in our docker compose file like so:

  prom:
    image: prom/prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    depends_on:
      - app

  graf:
    image: grafana/grafana
    privileged: true
    ports:
      - 3000:3000
    depends_on:
      - prom
    volumes:
      - ./graf-data:/var/lib/grafana # note: need `chmod 777 graf-data` to do this

Grafana is an open source graphing solution, commonly used with prometheus. I've included it here because it makes it much easier to digest and analyze the data once it gets in there. It's also just very commonly used with prometheus.

You will also need to set up your prometheus.yml file to find and scrape your application:

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
      - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['app:8080'] # run ifconfig and get host ip

The really important part of this config, once again leveraging the DNS in the docker compose file, is:

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
      - targets: ['app:8080'] # run ifconfig and get host ip

This tells prometheus to scrape our application every five seconds, and the URL to resolve to is our exposed /actuator/prometheus. To run the example provided in the source code, you will have to first create and modify a file to have the right permissions, then you can start it up:

$ mkdir graf-data && chmod 777 graf-data
$ docker-compose up -d

After everything downloads and comes up properly, you can get to grafana at port 3000 or prometheus on port 9090, and you should see the data flowing from our application to make it so.