Silencing

Reference documentation

• What is Sensu Silencing?
• How does silencing work?
• Silencing entry specification
• Examples
  • Silence all checks on a specific client
  • Silence a specific check on a specific client
  • Silence all checks on clients with a specific subscription
  • Silence a specific check on clients with a specific subscription
  • Silence a specific check on every client regardless of subscriptions
  • Deleting silencing entries
• Appendix: Deprecated stash-based silencing

What is Sensu Silencing?

As check results are processed by a Sensu server, the server executes event handlers to send alerts to personnel or otherwise relay event data to external services. Although event handlers can be directly configured with filters to improve overall signal-to-noise ratio, there are many scenarios in which operators receiving notifications from Sensu require an on-demand means to suppress alerts. Sensu’s built-in silencing provides the means to suppress execution of event handlers on an ad-hoc basis. By using a dashboard or other tool to interact with the /silenced API, operators can mute notifications on-the-fly.

The /silenced API manipulates silencing entries in the Sensu data store. These entries describe a combination of check name and subscription. When the check name and/or subscription described in a silencing entry match an event, the handler will not be executed and an informational message will be logged. Individual handlers may opt-out of silencing by configuring the handle_silenced attribute.

NOTE: Silencing described in this reference document is implemented in Sensu version 0.26 or later and Sensu Enterprise 2.0 or later.

When to use silencing

Silencing is used to prevent handlers from being triggered based on the check name present in a check result or the subscriptions associated with the client that published the check result. This can be desirable in many scenarios, giving operators the ability to quiet incoming alerts while coordinating their response.

Sensu silencing entries make it possible to:

• Silence all checks on a specific client
• Silence a specific check on a specific client
• Silence all checks on clients with a specific subscription
• Silence a specific check on clients with a specific subscription
• Silence a specific check on every client regardless of subscriptions

In addition to the above combinations, silencing entries support:

• Expiration after a specified number of seconds
• Expiration after check returns to OK state (resolves)
• Describing the “reason” or rationale
• Describing the “creator” or entity responsible for an entry

How does silencing work?

Silencing entries are created on an ad-hoc basis via the /silenced API endpoint. When silencing entries are successfully created via the API, they are assigned an ID in the format $SUBSCRIPTION:$CHECK, where $SUBSCRIPTION is the name of a Sensu client subscription and $CHECK is the name of a Sensu check. Silencing entries can be used to silence checks on specific clients by by taking advantage of per-client subscriptions added in Sensu 0.26, e.g. client:$CLIENT_NAME.

These silencing entries are persisted to the silenced registry in the Sensu data store. When the Sensu server processes subsequent check results, it consults this registry to determine whether or not a matching silencing entry exists. If one or more matching entries exist in the registry, the event context for the check result is updated to indicate that the event is silenced and include the ID of the entries which the check result matched.

When creating a silencing entry, a combination of check and subscription can be specified, but only one or the other is strictly required.

For example, when a silencing entry is created specifying only a check, its ID will contain an asterisk (or wildcard) in the $SUBSCRIPTION position. This indicates that any event with a matching check name will be marked as silenced, regardless of the originating client’s subscriptions.

Conversely, a silencing entry which specifies only a subscription will have an ID with an asterisk in the $CHECK position. This indicates that any event where the originating client’s subscriptions match the subscription specified in the entry will be marked as silenced, regardless of the check name.

NOTE: Starting with version 0.26, Sensu clients automatically add a subscription containing their client name prefixed with the string client:. For example, client i-424242 will automatically add subscription client:i-424242. Silencing checks at the individual client level requires clients to run Sensu 0.26+, or be manually configured with a client: subscription.

Silencing entry specification

Silencing entries are composed as a JSON document containing at least one of the required subscription or check attributes, and additional optional attributes as desired. Silencing entries are created, updated, and deleted by submitting JSON documents to endpoints on the /silenced API via HTTP POST as shown in the examples below.

Silencing entry attributes

{
  "check": "haproxy_status"
}

{
  "check": "haproxy_status",
  "subscription": "load_balancer"
}

{
  "subscription": "client:i-424242"
}

{
  "subscription": "client:i-424242",
  "check": "haproxy_status"
}

$ curl -s -X GET 127.0.0.1:4567/silenced | jq .
[
  {
    "expire": -1,
    "expire_on_resolve": false,
    "creator": null,
    "reason": null,
    "check": "mysql_status",
    "subscription": "appserver",
    "id": "appserver:mysql_status"
  }
]

{
  "expire": 3600,
  "check": "disk_utilization",
  "subscription": "client:i-424242"
}

{
  "expire_on_resolve": true,
  "check": "mysql_status"
}

{
  "creator": "Application Deploy Tool 5.0",
  "subscription": "appservers",
  "check": "app_status"
}

{
  "creator": "patrick",
  "subscription": "client:darkstar",
  "reason": "brb, rebooting"
}

"begin": 1512512023

Examples

Silence all checks on a specific client

Assume a Sensu client “i-424242” which we wish to silence any alerts on. We’ll do this by taking advantage of per-client subscriptions:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"subscription": "client:i-424242"}' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

The HTTP/1.1 201 Created response indicates our POST was successful, so we should be able to use GET to see the resulting entry:

curl -s -X GET 127.0.0.1:4567/silenced | jq .
[
  {
    "expire": -1,
    "expire_on_resolve": false,
    "creator": null,
    "reason": null,
    "check": null,
    "subscription": "client:i-424242",
    "id": "client:i-424242:*"
  }
]

PRO TIP: Due to performance considerations, silencing entries cannot match events using regular expressions. Instead, use client subscriptions to create easily-silenceable groups of clients or checks. For example, create a site-specific subscription to silence all events when performing maintenance at that site.

Now, imagine that we’d like to make this entry expire in 3600 seconds:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"subscription": "client:i-424242", "expire": 3600 }' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

If we query the list of silenced entries again, we can see the value of "expire" has changed from -1 to a value which decrements as time passes:

curl -s -X GET 127.0.0.1:4567/silenced | jq .
[
  {
    "expire": 3557,
    "expire_on_resolve": false,
    "creator": null,
    "reason": null,
    "check": null,
    "subscription": "client:i-424242",
    "id": "client:i-424242:*"
  }
]

Silence a specific check on a specific client

Following on the previous example, let’s silence a check named “check_ntp” on client “i-424242”:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"subscription": "client:i-424242", "check": "check_ntp"}' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

Now suppose I’d like to ensure this silencing entry is deleted once I’ve resolved the underlying condition it is reporting on:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"subscription": "client:i-424242", "check": "check_ntp", "expire_on_resolve": true}' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

The optional expire_on_resolve attribute used here indicates that when the server processes a matching check from the specified client with status OK, this silencing entry will automatically be removed from the registry.

When used in combination with other attributes (e.g. creator and reason), this provides Sensu operators with way to acknowledge that they have received an alert, suppressing additional notifications, and automatically clearing the silencing entry when the check status returns to normal.

Silence all checks on clients with a specific subscription

Assume a client subscription “appserver” which we wish to silence completely. Just as with our example of silencing all checks on a specific client, we’ll create a silencing entry specifying only the applicable subscription:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"subscription": "appserver"}' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

Silence a specific check on clients with a specific subscription

Assume a check “mysql_status” which we wish to silence, running on Sensu clients with the subscription “appserver”:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"subscription": "appserver", "check": "mysql_status"}' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

The HTTP/1.1 201 Created response indicates our POST was successful, so we should be able to use GET to see the resulting entry:

$ curl -s -X GET 127.0.0.1:4567/silenced | jq .
[
  {
    "expire": -1,
    "expire_on_resolve": false,
    "creator": null,
    "reason": null,
    "check": "mysql_status",
    "subscription": "appserver",
    "id": "appserver:mysql_status"
  }
]

Silence a specific check on every client

Assume we’d like to silence the “mysql_status” check on every client in our infrastructure, regardless of their subscriptions:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"check": "mysql_status"}' \
http://127.0.0.1:4567/silenced

HTTP/1.1 201 Created

The HTTP/1.1 201 Created response indicates our POST was successful, so we should be able to use GET to see the resulting entry:

$ curl -s -X GET 127.0.0.1:4567/silenced | jq .
[
  {
    "expire": -1,
    "expire_on_resolve": false,
    "creator": null,
    "reason": null,
    "check": "mysql_status",
    "subscription": null,
    "id": "*:mysql_status"
  }
]

Deleting silencing entries

Assuming we know the ID of a silencing entry, we can delete or clear it by via HTTP POST on the /silenced/clear endpoint:

$ curl -s -i -X POST \
-H 'Content-Type: application/json' \
-d '{"id": "appserver:mysql_status"}' \
http://127.0.0.1:4567/silenced/clear

HTTP/1.1 204 No Content

In this case the HTTP/1.1 204 No Content response indicates our POST was successful, meaning the silencing entry has been cleared (deleted) from the silenced registry.

Appendix: Deprecated stash-based silencing

Comparing stash-based and native silencing

Prior to Sensu 0.26 the ability to silence notifications was implemented in external libraries like sensu-plugin, using specially crafted Sensu API stashes.

Although silencing via stashes has not yet been removed from sensu-plugin, it is deprecated by both the native silencing described in this document, and planned changes in sensu-plugin itself.

Sensu’s new built-in or “native” silencing offers the following advantages over the stash-based silencing model which preceded it:

• Works for any type of handler (e.g. pipe, TCP, transport) or Sensu Enterprise integration.
• Works for any pipe handler regardless of language; no dependency on sensu-plugin or similar libraries.
• Handlers can opt out of silencing via configuration (see handle_silenced attribute.).
• Silencing can be applied to clients not yet registered with the system by targeting subscriptions instead of client names.
• Silencing entries can be automatically removed once the corresponding check returns OK
• Lower overhead - does not require forking a handler process to access the API

Migrating from stash-based silencing

For most operators, a browser-based dashboard like Uchiwa or the Sensu Enterprise Dashboard is the primary interface for silencing notifications on an ad-hoc basis. As of Uchiwa 0.18 and Sensu Enterprise Console 2.0, these dashboards now use the /silenced API in lieu of the /stashes API.

Even after upgrading both Sensu and the dashboard to take advantage of the /silenced API, handler plugins will continue to query the /stashes API and honor stashes under the silence path.

As a result, we recommend the following steps as part of any migration effort:

• Sensu Client should be upgraded so that clients will add client:$CLIENT_NAME to their subscriptions. NOTE: The client:$CLIENT_NAME subscription is required for native silencing to work at the individual client level.
• Sensu API and Server should be updated prior to upgrading Uchiwa or Sensu Enterprise Console.
• Any existing entries under /stashes/silence should be recreated via the /silenced API prior to upgrading Uchiwa or Sensu Enterprise Console.
• Any custom tooling which uses the /stashes/silence pattern should be updated to use the new /silenced API.
• All entries in /stashes/silence be deleted via the /stashes API before upgrading Uchiwa 0.18+ or Sensu Enterprise Console 2.0+.