Sensu Checks and Credentials

Along with every other every self respecting Sys Admin, I worry about security. The latest worry I had was around the way I supplied credentials for a sensu check to authenticate against a postgresql db.

The check in question is called check_postgres_alive.rb from the Sensu Community Plugins. The check requires you to pass the credentials and other sensitive information as arguments, which I’m really not a fan of.

Don’t use environment variables#

My first thought was to create environment variables with the various bits I need to authenticate (via the use of Puppet and Hiera), and then reference the variables in the command for the sensu check. The issue I faced with this is that after testing everything I could confirm it worked, but after rolling out the check it seemed the environment variables were being ignored and then check was erroring.

I hadn’t realised Sensu doesn’t spawn a bash shell /bin/false, meaning it wouldn’t read the variables set in /etc/profiles.d, rendering the variables useless.

Took me a little while to realise this wasn’t the way forward, and then started to look at other avenues.

Custom Attributes and Check Command Tokens#

Sensu’s answer to this problem is via Check Command Tokens and Custom Attributes, which are specified in the Sensu clients clients.json file and then the check itself.

As you can see in the example:

{
  "client": {
    "name": "i-424242",
    "address": "10.0.2.101",
    "environment": "production",
    "subscriptions": [
      "production",
      "webserver",
      "mysql"
    ],
    "mysql": {
      "host": "10.0.2.101",
      "port": 3306,
      "user": "app",
      "password": "secret"
    }
  }
}

You can create the custom attributes like above e.g. mysql, and state the necessary information.

Next you’d create a check that will run on the same host as the client config above.

{
  "checks": {
    "check_mysql_available": {
      "command": "check_mysql_active.rb -u :::mysql.user::: -p :::mysql.password::: -h :::mysql.host:::",
      "subscribers": [
        "production"
      ],
      "interval": 60
    }
  }
}

As you can see, you then specify the the custom attributes from the client config, and utilise check command tokens :::mysql.user::: as a way to reference these variables.

One thing to mention, you will want to redact the sensitive information, otherwise any sort of dashboard like Uchiwa will display it, making all this security conscious work pointless.

"redact": ["password", "user"]

This would go in your client.json config on your Sensu client.

Cheeky config file#

Another option is to use files that contain the sensitive information to load from, a good example would be the check-mysql-alive.rb check.

It allows you to reference the mysql ini file for the credentials as an option, as opposed to stating them in the command.

Conclusion#

I was ripping my hair out over this, however I thought I’d do a small write up for anyone else who might read this.

Do’s

• Custom Attributes/Check Command Tokens
• Config files e.g. yaml

Dont’s

• Environment variables
• Putting the creds in the plugin
• Passing throug the creds as parameters for the check