Sensu Server

Reference documentation

What is the Sensu server?

The Sensu server schedules and publishes check execution requests to client subscriptions (via a Publish/Subscribe model), and provides a scalable event processing platform for processing check results and monitoring events.

Sensu Core vs Sensu Enterprise

The Sensu server comes in two flavors: the open-source Sensu Core (via the sensu-server process), and Sensu Enterprise (via the sensu-enterprise process). To learn more about the differences between Sensu Core and Sensu Enterprise, please visit the Sensu website.

NOTE: Sensu Enterprise (i.e. the sensu-enterprise process) was designed to be a drop-in replacement for the Sensu Core server and API (i.e. the sensu-server and sensu-api). As such, any mention of “the Sensu server” in the Sensu documentation also applies to the sensu-enterprise process, for Sensu Enterprise users.

WARNING: as noted above, the sensu-enterprise process is designed to replace both of the Sensu Core sensu-server and sensu-api processes. Because Sensu Enterprise will load the same configuration as Sensu Core, it is important that the Sensu Core processes are stopped before starting Sensu Enterprise to avoid known conflicts and processing errors such as attempting to bind on the same ports, etc.

Check execution scheduling

Check execution scheduling is performed by a Sensu server (see Sensu server task election). Checks are scheduled by querying Sensu’s configuration for defined checks – excluding check with the attributes "standalone": true or "publish": false – and calculating when executions should occur based on their defined intervals.

Check scheduling algorithm & synchronization

Sensu uses an internal algorithm for determining a unique “cadence” for Sensu checks, which uniqueness is based on the check name and interval. This algorithm outputs a value in milliseconds which the Sensu server will use as an offset before the next check request should be published. In practice, this means that – assuming system clocks are in sync between disparate Sensu servers – check requests for a given check (based on the check name) will be published at the exact same time. The also means that in the event of a Sensu server restart and/or Sensu server task re-election (i.e. if a new Sensu server is elected to become responsible for check execution scheduling), check execution scheduling intervals will remain consistent.

In fact, because this algorithm is also shared by the Sensu client – which provides decentralized check execution scheduling in the form of standalone checks – a check defined on the Sensu server and a matching standalone check defined on a Sensu client should also stay in sync with each other (again assuming that system clocks are in sync, and the check names and intervals are consistent).

Event processing

The Sensu server provides a scalable event processor. Event processing involves conversion of check results into Sensu events, and then applying any defined event filters, event data mutators, and event handlers. All event processing happens on a Sensu server system.

The event processing workflow happens in the following order:

Event -> Filter -> Mutator -> Handler

NOTE: events that are filtered out (i.e. removed) by Sensu event filters will not continue to be mutated by event data mutators or event handlers.

Sensu’s event processing capabilities can be distributed among multiple Sensu servers in a Sensu cluster. For more information on configuring a Sensu cluster, please see Scaling Sensu (below).

Automated task election

The Sensu server processes (i.e. sensu-server and sensu-enterprise) are designed to scale horizontally (i.e. by adding systems). No additional configuration is required to run a cluster of Sensu servers, other than the location of the transport and data store. When Sensu servers start, they participate in an election process to automatically distribute tasks. A Sensu server may be elected for more than one task. A server task can only run on one Sensu server at a time and will automatically failover to another Sensu server in the event of a service failure or restart.

All Sensu servers in a Sensu cluster monitor the state of task execution on a 10-second interval, automatically electing a new Sensu server for a task if the current one hasn’t confirmed execution in more than 30 seconds.

Server tasks

In a Sensu server cluster, responsibility for a distinct set of tasks is distributed amongst members of the cluster. The tasks only run on one Sensu server at a time and will automatically failover to another Sensu server in the event of a service failure or restart. When adding Sensu servers to a cluster, restarting the existing Sensu servers in the cluster will force a redistribution of tasks.

  • Check request publisher. The Sensu server is responsible for publishing check requests to the transport for all configured checks. See check execution scheduling for more information.
  • Client monitor. The Sensu server is responsible for monitoring the client registry and creating client keepalive events for stale clients.
  • Check result monitor. The Sensu server is responsible for monitoring check results and creating TTL events for check results with expired check TTLs
  • Check result aggregation pruning. The Sensu server is responsible for monitoring check aggregates and pruning stale aggregate results.

To observe which Sensu server is currently responsible for one or more tasks, see API /info.

Scaling Sensu

Coming soon…