Guest Post

How to use Slack and Fluentd to demonstrate social output

Phil Wilkins provides step-by-step instructions on how DevOps teams can enable Slack to capture log events from the open source data collection tool Fluentd.

Slack has become a leading team messaging collaboration tool with a strong API layer. It includes a free version that is simply limited by the size of conversation archives. As a cloud service as well, it makes for a great tool to illustrate the intersection of Fluentd and actionable log events through social platforms. While the following steps are specific to Slack, the principles involved here are the same for Microsoft Teams, Jabber and many other collaboration services.

If you are already using Slack, it is tempting to use an existing group if you want to run the example. However, you risk irritating other users by filling your channels with test messages. If you aren't using Slack, that's not a problem, a Slack account is easy to configure and use. During the configuration of Slack, ensure you keep a note of the API token -- recognizable by having a prefix of xoxb.

Slack provides a rich set of configuration options for the interaction. Unlike the MongoDB and many IaaS and PaaS-level plugins, Slack is a SaaS service, so the resolution of the Slack instance is both simplified and hidden from us by using a single token (no need for server addresses, etc.). The username isn't about credentials, rather how to represent the bot that the Fluentd plugin behaves as, therefore using a meaningful name is worthwhile. The channel relates to the Slack channel in which the messages sent will be displayed. The general channel exists by default, but if you want to control who sees the messages, then you may wish to create a custom channel in Slack and restrict access to that channel; otherwise everyone within your enterprise Slack setup will see every operational message.

Unified Logging with Fluentd book coverClick the image to learn
more about the book.
Take 35% off any format
by entering ttwilkins into
the discount code box
at checkout.

The message and message_keys attributes work together, with the message using %s to indicate where the values of the identified payload elements are inserted. The references in the message relate to the names of the JSON payload elements listed in the message_keys in order sequence.

The title and title_keys work in the same manner as the message and message_keys but for the title displayed in the Slack UI with that message. In our case, we're just going to use the tag. The final part is the flush attribute --  this tells the plugin how quickly to push the Slack messages to the user. Multiple messages can be grouped if the period is too long. To keep things moving quickly, let's flush every second.

Edit the existing configuration provided (Chapter4/Fluentd/rotating-file-read-slack-out.conf) to incorporate the details captured, in the Slack setup. This is illustrated in Listing 1.

Listing 1. /Chapter4/Fluentd/rotating-file-read-slack-out.conf – match configuration

<match *>
   @type slack
   token xoxb-9999999999999-999999999999-XXXXXXXXXXXXXXXXXXXXXXXX    #A
   username UnifiedFluent    #B
   icon_emoji :ghost:    #C
   channel general    #D
   message Tell me if you've heard this before - %s    #E #G
   message_keys msg    #F #G
   title %s    #H #I
   title_keys tag    #I
   flush_interval 1s    #J
</match>

#A: This is where to token obtained from Slack is put, to correctly identify the workspace and legitimize the connection.
#B: Username that will be shown in the Slack conversation.
#C: Define an individual emoji (icon) to associate the bot.
#D: Which channel to place the messages into, based on the name. In our demo, we're just using the default channel.
#E: Message to be displayed, referencing values in the order provided in the config.
#F: Name the log event's JSON elements to be included in the log event.
#G: These attributes work as a pair.
#H: Title for the message takes the values from title_keys. Using order to map between the config values.
#I: Definition of the message's title.
#J: Defines the frequency at which Fluentd will get Slack to publish the log events.

Before we start running the solution, we also need to install the Slack plugin with the command fluent-gem install fluent-plugin-slack. Once installed, we can start up the log simulator and Fluentd with the following commands:

  • fluentd -c Chapter4/Fluentd/rotating-file-read-slack-out.conf
  • groovy LogSimulator.groovy Chapter4/SimulatorConfig/social-logs.properties ./TestData/small-source.txt

Once started, if you open the #general channel in the web client or app, you will see messages from Fluentd flowing through.

Figure 1. Our Fluentd log events being displayed in Slack
Figure 1. Our Fluentd log events being displayed in Slack

All the details for the Slack plugin can be obtained from Github. Our illustration of Slack use is relatively straightforward. Through the use of several plugins, we could very easily go from the source tag to addressing the Slack messages to the most relevant individual or group directly, or having different channels for each application and directing the messages to those channels.

Security best practices for handling tokens and credentials

Good security practice has told us we should not hardwire credentials into plain text files, as anyone can look at the file and get sensitive credentials. If you commit to Github a configuration file or code which contains a string that looks like a security token for a service Github knows about, they will flag it with the service provider. When Slack is told and decides it is a valid token, then be assured that they will revoke the token. 

So how do we address such a problem? Depending on your circumstances, a couple options include:

  • Set the sensitive credentials into environment variables with a limited session scope -- the environment variable can be configured several ways, such as using tools like Che and Puppet to set up the values from a key store.
  • Embed into the application the means to access a Keystore or a common security secrets management solution such as Hashicorp Vault.

While the configuration files may not look like it based on what we have seen so far, we can achieve both of these approaches within a Fluentd configuration file as Fluentd allows us to embed Ruby fragments into the configuration. This doesn't mean we need to immediately learn Ruby. For the first of these approaches, we just need to understand a couple of basic patterns. The approach of embedding calls to Vault is more challenging, so we'll address that when we talk about custom plugin development.

The notation to embed the Ruby code is to prefix the Ruby code with "#{ and close it with }". Ruby's core language provides us with a core set of classes and static operations, giving a lot of access to information such as environment variables.  Environment variables can be accessed using the ENV class (https://ruby-doc.org/core-2.5.0/ENV.html). So, if we created an environment variable called slack-token, rather than including the token directly in the configuration, we could replace the setting of the attribute with the following:

   token "{ENV["slack-token"]}"

Externalizing Slack configuration attributes: An exercise

Exercise: Set up your environment so that you have an environment variable called SlackToken which is set to hold the token you have previously obtained. Then customize Chapter4/Fluentd/rotating-file-read-slack-out.conf to use the environment variable, and re-run the example setup with the commands:

  • fluentd -c Chapter4/Fluentd/rotating-file-read-slack-out.conf
  • groovy LogSimulator.groovy Chapter4/SimulatorConfig/social-logs.properties ./TestData/small-source.txt

Confirm that log events are arriving in Slack.

The right tool for the right job

The first chapter of the book Unified Logging with Fluentd highlights the issue of different people wanting to use different tools for a range of reasons, such as:

  • To perform log analytics, as different tools have different strengths and weaknesses;
  • Multi-cloud, so specialist teams (and cost considerations of network traffic) mean using different cloud vendor tools.
  • Decisions influenced individual preferences and politics -- previous experience, etc.

As we have illustrated, Fluentd can support a lot of social platforms and protocols. Of course, this wouldn't be the only location for log events to be placed. One of the core types of destination is a log analytics tool or platform. Fluentd has a large number of plugins to feed log analytics platforms. In addition to the two we initially mentioned, other major solutions that can be easily plugged in include:

  • Azure Monitor
  • Graphite
  • Elasticsearch
  • Cloudwatch
  • Google Cloud (Stackdriver)
  • Sumologic
  • Logzio
  • Oracle Log Analytics

We can also send logs to a variety of data storage solutions, either to hold for later use or perform data analytics with -- for example:

  • Postgres, InfluxDB, MySQL, Couchbase, DynamoDB, AeroSpike, SQL Server, Cassandra;
  • Kafka, AWS Kenesis (time series store, event streaming); and
  • Storage areas such as AWS S3, Google Cloud Storage, Google BigQuery, WebHDFS.

So, the question becomes, what are your needs and which tool(s) is the best fit? If your requirements change over time, then you add or remove your targets as needed. Changing the technology is probably going to raise more challenging questions about what to do with existing log events, not how to get the data into the solution.

About the author
Phil Wilkins has spent over 25 years in the software industry. He works as a technology evangelist for Capgemini, specializing in cloud integration, APIs and related technologies. Phil is TOGAF certified, and recognized by Oracle as an Ace Director (independent technology advocate) for his contributions to the integration and PaaS community.

Dig Deeper on IT systems management and monitoring

Software Quality
App Architecture
Cloud Computing
SearchAWS
TheServerSide.com
Data Center
Close