ForgeRock, Prometheus & Grafana integration (Part 1)

6 min readApr 19, 2019

Today, Prometheus and Grafana is one of the most used monitoring stack by DevOps Teams. Since a few version now, ForgeRock platform components provide with Prometheus endpoints and more and more customers are using them. In this post I’ll give an introduction of the integration of ForgeRock Identity Platform with Prometheus and Grafana.

What are Prometheus and Grafana?

Prometheus and Grafana is a monitoring stack composed by two different components which can be used separately and have some overlapping functionalities. This stack is becoming more and more used by DevOps Teams especially because it is easy to deploy and use. The following figure is a rapid presentation of Prometheus & Grafana.

Note: You can navigate to their web sites to get a more detailed presentation of Prometheus and Grafana.

How to use it with ForgeRock?

For all the components of our current ForgeRock Platform we provide with HTTP/HTTPS endpoints to be used to gather metrics. These endpoints are ment to be polled regularly by Prometheus (you just have to configure Prometheus to get these datas). Then Prometheus stores it locally and you can define queries or variables to enrich the data. Finally, you use Grafana to create Dashboard and alerts. For all the components of Forgerock Platform we created Dashboard example to import in Grafana. The following figures shows a high level architecture.

Note: The communication between Prometheus and Grafana is not possible OOTB with SSL/HTTPS, you have to setup a reverse proxy in between.

ForgeRock Prometheus Endpoints

All ForgeRock components can be polled by Prometheus as long as you activate it. A Prometheus Endpoint is a simple URL (Protected or not) you can access to get metrics information. For instance with a ForgeRock Directory Server, you can access the endpoint with this URL (default configuration): http(s)://<DS ServerName>:<PortNumber>/metrics/prometheus where <DS ServerName> is equal to the server where the endpoint is available and <PortNumber> is equal to the port define (most likely 8080 or 8443 depending if we are using HTTP or HTTPS).

When you access the endpoint using a browser the metics page should look like this :

... # HELP ds_backend_db_cache_misses_leaf_nodes Number of leaf nodes (data records) requested by btree operations that were not in the database cache
# TYPE ds_backend_db_cache_misses_leaf_nodes gauge
ds_backend_db_cache_misses_leaf_nodes{backend="dsEvaluation",type="db",} 129.0
# HELP ds_backend_db_cache_size_active_bytes Size of the database cache
# TYPE ds_backend_db_cache_size_active_bytes gauge
ds_backend_db_cache_size_active_bytes{backend="dsEvaluation",type="db",} 3275885.0
# HELP ds_backend_db_cache_total_tries_internal_nodes Number of internal nodes requested by btree operations
# TYPE ds_backend_db_cache_total_tries_internal_nodes gauge
ds_backend_db_cache_total_tries_internal_nodes{backend="dsEvaluation",type="db",} 88047....

A line contains basically : the metric name, a few variables (backend, metric type, etc..) and a value (equals to the value of the metric). There are 4 types of metrics : Counter, Gauge, Summary and Timer. I suggest you to read this ForgeRock page to get explanations on these different types.

Activate Prometheus Endpoint in ForgeRock

The Prometheus endpoint activation in ForgeRock components is straightforward.

ForgeRock Access Management (AM)

Connect to AM web console and follow theses steps:

Activate the monitoring globally: go to Configure > Global services > Monitoring and ensure that Monitoring Status is enabled.
The monitoring service must be enable globally to be able to activate the Prometheus endpoint which is considered as a Secondary Configuration.
Add Prometheus as secondary monitoring configuration: go to Configure > Global services > Monitoring, select Secondary Configurations, clic Add a Secondary Configuration and select Prometheus Reporter.
Activate Prometheus and add authentication to the endpoint: On the next screen Enable the service, select HTTP Basic Authentication type and define a Username and Password.
Note: this username / password will be used by Prometheus to access the page.

That’s all you got to do to enable Prometheus endpoint in AM. To check it you may access the following URL with your favorite web browser http(s)://<AM InstanceName>:<AM port>/am/json/metrics/prometheuswhere <AM InstanceName> is equal to the AM instance server name where the endpoint is available and <AM port> is equal to the port number where AM is available (it may be 8080 for HTTP or 8443 for HTTPS).

ForgeRock Identity Management (IDM)

In IDM the Prometheus endpoint is already configured and enabled. An authentication module is defined in <IDM-SETUM-PATH>/conf/authentication.json file. When Prometheus is accessing the endpoint, this module authenticates Prometheus using the login and password defined in the file <IDM-SETUP-PATH>/resolver/boot.properties (see a part of this file below).

...10:  # Prometheus endpoint authentication
11:  openidm.prometheus.username=prometheus
12:  openidm.prometheus.password=prometheus...

By default, the Prometheus endpoint is available at this URL : http(s)://<IDM InstanceName>:<IDM port>/idm/metrics/prometheuswhere <IDM InstanceName> is equal to the IDM instance server name where the endpoint is available and <IDM port> is equal to the port number where IDM is available (it may be 8080 for HTTP or 8443 for HTTPS).

ForgeRock Directory Server (DS)

Connect to your DS host with a shell, navigate to the DS bin directory <DS-SETUP-PATH>/binand start the dsconfig command.

Manage HTTP endpoints: after connecting to your DS instance interactively with dsconfig, enter 20 to select HTTP Endpoint,
Create new HTTP endpoint: on the next menu, enter 1 to select Create a new HTTP endpoint.
Note: during this step, you may enter 4 to List existing HTTP Endpoints to check if there isn’t an existing prometheus endpoint enabled already.
Create a Prometheus Endpoint: on the next menu, enter 6 to create a Prometheus HTTP Endpoint.
Configure your Prometheus Endpoint URL: on this menu you have to define the URL path of the endpoint. This is an absolute path from the root URL and must include a slash at the beginning. For instance if you enter /metrics/prometheusthen, your endpoint will be available at the following URL: http(s)://<DS ServerName>:<PortNumber>/metrics/prometheus where <DS ServerName> is equal to the server where the endpoint is available and <PortNumber> is equal to the port define (most likely 8080 or 8443 depending if we are using HTTP or HTTPS).
Note: depending on your DS setup, it is possible that the /metrics/prometheus is defined already.
Note 2: don’t forget the initial / when defining the URL.
Select authentication mechanisms for the endpoint: on this menu you can select one or many different authentication methods to access the URL. The easiest is HTTP Basic, enter 2 to select this method and then enter 1 to enable the endpoint.
Check and start the endpoint: at this step you get a summary, just enter f to finish and start the Prometheus endpoint.

That’s it, Prometheus endpoint is available in DS. To check it, you can access the following URL with your web browser http(s)://<DS ServerName>:<PortNumber><HTTPPromEndpoint> where <DS ServerName> is equal to the server where the endpoint is available, <PortNumber> is equal to the port define (most likely 8080 or 8443 depending if we are using HTTP or HTTPS) and <HTTPPromEndpoint> is equal to the path you defined in step #4.

ForgeRock Identity Gateway (IG)

The Prometheus endpoint for IG is available out of the box, no additional configuration nor activation is needed to work. However, unlike the other components of ForgeRock Identity platform, this endpoint is not protected and can be accessed by everybody (no special credential are needed).

Note: this Identity Gateway documentation : Protecting the Monitoring Endpoints explains how to secure the end point with Basic Authentication in 2 steps.

Available Metrics

ForgeRock platform provides with a great number of metrics for all the components. These metrics are really well documented, so I will just point you out to the documentation for each product.

AM metrics documentation: https://backstage.forgerock.com/docs/am/6.5/maintenance-guide/#sec-maint-monitoring-metrics
Note: this documentation is valid for all metrics tools supported by AM, not only Prometheus.
IDM Prometheus metrics documentation: https://backstage.forgerock.com/docs/idm/6.5/integrators-guide/#prometheus-metrics
DS Prometheus metrics documentation: https://backstage.forgerock.com/docs/ds/6/reference/#monitoring-metrics-prometheus
IG Prometheus metrics documentation: https://backstage.forgerock.com/docs/ig/6.5/reference/#prom-route-metrics-prom & https://backstage.forgerock.com/docs/ig/6.5/reference/#prom-router-metrics-prom

Note: In addition to these standard metrics it is possible to add custom ones. For instance in Authentication trees you can create timers or probes to analyse the performance of your tree (I’ll discuss this topic in another blog post).

Conclusion (Part 1)

With this post I just introduced how ForgeRock, Prometheus and Grafana are integrated and especially how the ForgeRock Platform provides Prometheus endpoints. In the next part I will explain how to configure Prometheus with the prometheus.yml file and how to use the Grafana dashboard examples ForgeRock provides for each component.