Monitoring Sensu with Sensu

In this guide, we’ll walk through best practices and strategies for monitoring Sensu with Sensu. By the end of the guide, you should have a thorough understanding of what is required to ensure your Sensu components are properly monitored, including:

In order to completely monitor a Sensu stack (Sensu server, Sensu API, Redis, RabbitMQ), you will need to have at least one other independent Sensu. A single Sensu stack cannot monitor itself completely, as if some components are down, Sensu will not be able to create events. As Sensu plugins are used in this guide, installing plugins has more information on how to install those.

NOTE: This guide assumes you are not using Sensu clustering, RabbitMQ clustering, or Redis Sentinels. You can still monitor each server using the strategies described in this guide, but note that in order to effectively monitor clustered instances, you’ll need to employ a different methodology.

Monitoring Sensu

Monitoring Sensu Server

The host running the sensu-server service should be monitored in two ways:

  • Locally by a sensu-client process for operating system checks/metrics and Sensu services like Uchiwa that are not part of the Sensu event system.
  • Remotely from an entirely independent Sensu stack to monitor pieces of the Sensu stack that if down, will make Sensu unable to create events.

Monitoring Sensu Server Locally

Monitoring the host that the sensu-server process runs on should be done like any other node in your infrastructure. This includes, but is not limited to, checks and metrics for CPU, memory, disk, and networking. You can find more plugins at the Sensu Community Homepage.

Monitoring Sensu Server Remotely

To monitor the sensu-server server process, you will need to do so from an independent Sensu stack. This can be done by reaching out to Sensu’s API health endpoint and using the check-http plugin with the following check definition.

{
  "checks": {
    "check_sensu_server_port": {
      "command": "check-http.rb -h remote-api-hostname -P 4567 -p /health?consumers=1 --response-code 204",
      "subscribers": [
        "monitor_remote_sensu_api"
      ],
      "interval": 60
    }
  }
}

Monitoring Sensu API

To monitor the sensu-api service, you will need to do so from an independent Sensu stack. This can be done by reaching out to the port that the API is listening on using the check-port plugin with the following check definition.

{
  "checks": {
    "check_sensu_api_port": {
      "command": "check-ports.rb -H remote-sensu-server-hostname -p 4567",
      "subscribers": [
        "monitor_remote_sensu_api"
      ],
      "interval": 60
    }
  }
}

Monitoring Uchiwa

Method 1: Monitor Uchiwa with a process check

To monitor the Uchiwa Dashboard, you will need to check for two processes named uchiwa using the check-process plugin with the following check definition. This check will return a check result with status 2 if less than two processes are running with the string /opt/uchiwa/bin/uchiwa. We look for two processes because the uchiwa service has a parent and child process.

{
  "checks": {
    "check_uchiwa_process": {
      "command": "/opt/sensu/embedded/bin/check-process.rb -p /opt/uchiwa/bin/uchiwa -C 2",
      "subscribers": [
        "monitor_local_uchiwa_processes"
      ],
      "interval": 60
    }
  }
}

Method 2: Monitor Uchiwa with a remote network port check

You can also monitor the uchiwa Dashboard with a remote port check using the check-port plugin with the following check definition.

{
  "checks": {
    "check_uchiwa_port": {
      "command": "check-ports.rb -H uchiwa-remote-hostname -p 3000",
      "subscribers": [
        "monitor_remote_uchiwa_port"
      ],
      "interval": 60
    }
  }
}

PRO TIP: Use both methods for complete monitoring of Uchiwa. This way, you are able to catch when the processes are not running and when a firewall port is not open.

Monitoring RabbitMQ

RabbitMQ has a management plugin that must be enabled to allow the Sensu RabbitMQ check and metric plugins to function properly. You can find instructions for enabling it in RabbitMQ’s management plugin docs.

You will also need an administrative level user that has full access to your /sensu virtualhost. Below is an example of how to create a monitor user using the rabbitmqctl command line tool.

NOTE: You can use the same RabbitMQ user that Sensu uses if you do not want to create another user, as they are role and permission equivalent. This guide will continue to use the monitor_user in examples.

rabbitmqctl add_user monitor_user password
rabbitmqctl set_user_tags monitor_user monitoring
rabbitmqctl set_permissions -p /sensu monitor_user "" "" ".*"

To monitor the RabbitMQ instance, you will need to do so from an independent Sensu stack. The rabbitmq-alive plugin provides that ability using the following configuration with your RabbitMQ credentials.

{
  "checks": {
    "check_rabbitmq_alive": {
      "command": "check-rabbitmq-alive.rb -w remote-rabbitmq-host -v %2Fsensu -u monitor_user -p password -P 15672",
      "subscribers": [
        "monitor_remote_rabbitmq"
      ],
      "interval": 60
    }
  }
}

While Sensu does not provide benchmarks for healthy RabbitMQ keepalives and results queues, you can use the metrics-rabbitmq-queue plugin to establish a baseline for what looks normal for your environment.

{
  "checks": {
    "collect_rabbitmq_keepalives_queue": {
      "command": "metrics-rabbitmq-queue.rb --user monitor_user --password password --vhost /sensu --host localhost --port 15672 --filter keepalives",
      "subscribers": [
        "rabbitmq_keepalive_queue"
      ],
      "interval": 60,
      "type": "metric"
    }
  }
}
{
  "checks": {
    "collect_rabbitmq_results_queue": {
      "command": "metrics-rabbitmq-queue.rb --user monitor_user --password password --vhost /sensu --host localhost --port 15672 --filter results",
      "subscribers": [
        "rabbitmq_results_queue"
      ],
      "interval": 60,
      "type": "metric"
    }
  }
}

Then you can use the check-rabbitmq-check plugin to create checks for both the keepalives and results queues based on your benchmarks. The following check definition uses 250 as the normal depth for both queues.

{
  "checks": {
    "check_rabbitmq_queue": {
      "command": "/opt/sensu/embedded/bin/check-rabbitmq-queue.rb --username monitor_user --password password --vhost /sensu --port 15672 --queue keepalives,results -w 500 -c 1000",
      "subscribers": [
        "monitor_rabbitmq_keepalive_queue"
      ],
      "interval": 60
    }
  }
}

Monitoring Redis

To monitor the Redis instance, you will need to configure Redis to bind to an interface that your independent Sensu stack can reach and open TCP port 6379. Then you can use the check-redis-ping plugin to monitor whether Redis is responsive using the following check definition.

{
  "checks": {
    "redis_ping": {
      "command": "check-redis-ping.rb -h remote.redis.hostname",
      "subscribers": [
        "monitor_remote_redis"
      ],
      "interval": 60
    }
  }
}