Tech - Check your Salt - Part 3

Overview

In the previous post, we saw how beacons were deployed in all machines using state files. Salt internally uses zero MQ messaging for communication between the master and minions. As with any messaging framework, there are publisher(s) and consumer(s). One can think of the beacons (on salt minions) as publishers and reactors (on salt master) as consumers.

Salt Concepts Covered

In this post, the following Salt concepts are covered -
  • Beacons - Monitor minions for state changes, system diagnostics etc.
  • Reactors - Trigger reactions when events occur in the managed systems running minions.

Beacons

A beacon on the minion can be used to monitor various system events such as - memory usage, CPU load average, overall statistics, user commands etc. The beacon deployed in the previous post contains the 'sh' module. Sh module monitors all commands run on a machine which has the salt minion running.

Note: Every time a beacon is deployed to the salt minions, the minions needs to be restarted.

Reactors

Now that we have the minion setup, they keep sending the events to the salt master. We now create a reactor that will respond to these events.

Create the /etc/salt/master.d/reactor.conf file on the master -

# contents of the reactor.conf file
reactor:
  - 'salt/beacon/*/sh/*'
  - '/srv/reactor/sh.sls'

Create a Slack account and add an application in it. Use the API key and the channel name from Slack in the below reactor - sh.sls.

Next, create the /srv/reactor/sh.sls on the master -

Send sh to Slack:
  local.slack.post_message:
    - tgt: {{ data['id'] }}
    - kwarg:
         channel: "[Slack Channel Name]"
         api_key: "[Slack API Key]"
        message: "{{ data['id'] }} : user {{ data['user'] }} ran command {{ data['cmd'] }} with arguments {{ data['args']  }}."
        from_name: "Sh Bot"

Note: Every time a new reactor is deployed to the salt master, the master needs to be restarted.

After the beacons and reactors have been set up, you can also monitor the events using the command (in the salt master) - 'sudo salt-run state.event pretty=True'

Now, all commands that are run on the salt minions can be viewed in Slack (both on a the desktop and mobile apps). This way, the user logins, commands run etc. across the infrastructure can now be monitored and tracked using a Slack channel.

Slack Desktop




Slack Mobile



Summary

So far, we have seen how salt has been used to -
  1. Ease the management of infrastructure (servers, desktops, IoT devices)
  2. Maintain consistent state across the infrastructure environment
  3. Proactively respond to events in the infrastructure for monitoring and compliance purposes

Comments

Popular posts from this blog

Meta Prompting for Success

Book Review - The Price of Being Fair

How to create a big picture