Elastic Stack Security tutorial: How to create detection rules

This excerpt from 'Threat Hunting with Elastic Stack' provides step-by-step instructions to create detection rules and monitor network security events data.

Security researchers have a variety of threat hunting tools at their disposal. One such tool worth considering is the free, open code Elastic Stack, said Andrew Pease, principal security research engineer at Elastic and author of Threat Hunting with Elastic Stack.

In his book, Pease introduces readers to the Elastic Stack suite. He explains how the suite of tools helps users collect data, analyze it and create visualizations.

In this excerpt from Chapter 8, "The Elastic Security App," Pease explains how to start using different functions of the Elastic Stack Security app, formerly Elastic SIEM. Learn what to expect from the Security dashboard, such as how to create and implement detection rules and exceptions and how to monitor them.

Download a PDF of Chapter 8 to learn how to create different machine learning rules, how to use and customize the trend timeline dashboard page, and how to understand the event details panel in the Security app.

The Elastic Security app overview

The Elastic Security app is the central point for Elastic's security solution. It includes a security news feed, host and network data, detections, timelines, cases, and an abstracted view into the administration of the Elastic endpoint configuration.

To get to the Elastic Security app, click on the hamburger menu and select Overview under the Security heading. This landing page will show you the highlights of the events that are in the security app. From here we can jump into specific sections that show their relevant data:

Screenshot of the Elastic Stack Security app main dashboard
Figure 8.1 -- Security app overview

You can scroll through this Overview section to get a high-level look at the different types of data that are reflected in the app. Most notably, at the bottom of the Overview page, there is a breakdown of the different datasets, separated by host and network, that we're sending into the Elastic Stack:

Screenshot of Elastic Stack Security app main dashboard bottom
Figure 8.2 -- Security app datasets

This Overview page allows us to see important information that is relevant across the entire Security app. To get additional information, we will use the section tabs at the top of the Overview page.

Screenshot of book cover for 'Threat Hunting with Elastic Stack'Learn more about Andrew Pease's
Threat Hunting with Elastic

In this section, we learned how to get to the Elastic Security app Overview page. Next, we'll explore the detection engine.

The detection engine

The Detections section is used to investigate and create detection logic. Detection logic can be the results of a malware hit from a signature or behavior as well as potentially malicious activity. As of Elastic version 7.12, the detection engine has over 500 pre-built rules that are created by the Elastic Intelligence and Analytics Team and the Elastic community:

Screenshot of Elastic Stack Security app Detections tab
Figure 8.3 -- Detection section

Keeping with the theme across the rest of Kibana, you can apply specific queries directly into the Detections section or apply a date picker selection to narrow any searches.

In this section, we began to explore the detection engine of the Elastic Security app. In the next section, we'll learn about managing the detection rules.

Managing detection rules

One of the most powerful features of the Elastic Security app is the detection rules. Detection rules are pre-configured queries that compare events from various data sources to identify non-signature-based malicious activity.

As an example, perhaps you want to know whether you have systems that are receiving Remote Desktop Protocol (RDP) connections from outside your network, whether someone is trying to brute force through Secure Shell (SSH), or someone is trying to export your Windows Registry Hive. These things could be malicious in your environment, but these are events that will not be detected by traditional anti-virus.

Elastic has released hundreds of open source rules for the detection engine and has made them all available on GitHub. As I mentioned before, there are 546 rules available for free. Not only are the rules available on GitHub, but they are also automatically loaded in Kibana. You may remember in Chapter 5, Building Your Hunting Lab -- Part 2, we loaded all of the prebuilt rules:

Screenshot of Elastic Stack Security app Detection alerts view
Figure 8.4 -- Manage detection rules

If you click on the Manage detection rules button, it will open the Detection Rules management section.

From here, we can see the Rules, Rules Monitoring, and Exception Lists tabs.


From this tab, we can enable different rules, search for rules by their names or tags, or dig into the rules to learn more about them:

Screenshot of Elastic Stack Security app Detection rules
Figure 8.5 -- Rules overview

Clicking on a rule will open the rule so that you can inspect the metadata about the rule, where the data must come from, what the query is, and so on:

Important note

You cannot modify the Elastic-provided rules, but you may make a duplicate and modify the duplicate if necessary.

Screenshot of the Elastic Stack Security app Detection rules example
Figure 8.6 -- Public IP Reconnaissance Activity network rule

This overview of the rules will help you determine what rules you want to enable and what rules don't make sense in your environment. As an example, if you aren't collecting cloud security rules, or Linux or macOS events, it doesn't make sense to enable those rules.

Rule monitoring

Clicking on the Rule Monitoring tab will give you a view of the amount of time the rules take to run:

Screenshot of Elastic Stack Security app Detection rules monitoring
Figure 8.7 -- Rule Monitoring

Rule Monitoring can be helpful if you're noticing a performance impact; you can look at what rules are taking the most time and decide if you need to increase the resources for your stack or if those rules are needed.

Exception Lists

The Exception Lists tab is where you can view any exceptions you've created for rules or the endpoint:

Screenshot of Elastic Stack Security app Detection rules exception lists
Figure 8.8 -- Exception Lists

We'll talk more about the exception framework in the Event actions section, a bit further on in the chapter.

Creating a detection rule

I mentioned that Elastic provides 546 rules for you, but we can also create rules that fit a specific threat profile for our environment.

Rules can either be created using a Python module that Elastic provides and has made available, or be created and made available through Kibana.

To get started, click the blue Create new rule button:

Screenshot of a rule in the Elastic Stack Security app
Figure 8.9 -- Create new rule

There are five types of rules that you can create:

  • A Custom query (KQL or Lucene rule)
  • A Machine Learning rule
  • A Threshold rule
  • An Event Correlation rule
  • An Indicator Match rule

Next, we'll walk through the creation of these rule types.

Important note

For the detection rules, the first section (Define rule) will change depending on the rule type that you're going to use, but the three follow-on sections will all be the same (About rule, Schedule rule, and Rule actions). We'll go all the way through the four sections for the Custom query type and just the first section for the other four rule types.

Custom query rule

By default, the rule type will be Custom query. This is how you'd create a KQL or Lucene query.

Below the rule type, you can select what index patterns your data will be in. By default, all of the possible index patterns are added. As we only have Endpoint, Packetbeat, and Winlogbeat data, we can safely remove the unused datasets.

Next, we can enter our query. As an example, I am looking for network connections to the domain packtpub.com:

Screenshot of creating a new rule in Elastic Stack Security app
Figure 8.10 -- Custom Query

Here we can see what the first part of the rule will look like.

Elastic has added the Preview results feature so you can test to see whether your query is working as intended:

Screenshot of preview mode of Elastic Stack Security app Detection rule
Figure 8.11 -- Preview results

After we've set up the rule type and the data sources, written our query, and tested the results, we can click on Continue to move onto the next step.

In the second section, we can define the name, description, severity, and risk score, and add any organization tags:

Screenshot of preview mode of Elastic Stack Security app Detection rule description
Figure 8.12 -- Rule description

The default severity is Low and the risk score is 21. As you change the severity, the risk score will automatically adjust. This can also be manually tuned if you have organizational policies that dictate a severity and risk matrix.

You can also override the defaults for severity and risk. In the preceding figure, I have changed the severity from Low to Critical if the network traffic is unencrypted over the HTTP protocol instead of TLS.

Clicking on the Advanced settings dropdown, we can add some additional information about the rule. These settings are optional:

Screenshot of Elastic Stack Security app Detection rule advanced settings
Figure 8.13 -- Advanced settings

Here we can define any reference URLs that can help provide some context when performing an investigation. I like to put the source of the threat reports that may have led to the rule creation here.

We can provide some false positive examples.

We can add MITRE ATT&CK tactics, techniques, and subtechniques. The ATT&CK model was covered in Chapter 1, Introduction to Cyber Threat Intelligence, Analytical Models, and Frameworks.

We can also add investigative notes. This can be used to link to organization-specific documentation, points of contact, and so on. This guide renders GitHub-flavored Markdown, which is helpful to use for inserting hyperlinks and tooltips.

GitHub-flavored Markdown can create hyperlinks using [URL Text](url) and tooltips with !{tooltip[anchor text](helpful description)}.

Moving on, we can populate the Author field and add the appropriate License details:

Screenshot of Elastic Stack Security app Detection alert sorting
Figure 8.14 -- Detection alerts trend sorting

We can also choose to apply any exceptions to this rule; we don't have any yet, but if we wanted to apply exceptions, we could check that box.

Building block rules are low-risk rules that we can create that will still write to the rules indices but not be displayed in the Alerts table in the main Detections view. This is helpful if you want to create a rule for context for other rules but not clutter up your view.

Rule name override will allow you to override the rule name we set before with the value of a field from the event. As an example, we could use the destination.domain field to name this rule packtpub.com when it is displayed in the Alerts table.

If we want to use a different timestamp than the default @timestamp, we can define that here.

Next, we can click Continue to move on.

We can define the schedule for the rule. This is how often the rule will run and how far back it should look. The lookback is to ensure there aren't specific events that happen to fall between the rule executions that could be missed.

The default is to run every 5 minutes with a 1-minute lookback. I prefer to change this to run every 9 minutes with a 1-minute lookback:

Screenshot for scheduling when to run rules in Elastic Stack Security app
Figure 8.15 -- Schedule the rule

After we have set the schedule and the lookback, we can click Continue to move on to Rule actions.

We can define how often the rule actions are performed, but they allow you to send notifications to third-party services. Actions to these external services are a paid feature.

Finally, we can create the rule. I always prefer to create the rule without activating it (meaning it will generate an event in the Detections section) so I can take one final look at the completed rule before I run it:

Screenshot of creating a custom rule in Elastic Stack Security app
Figure 8.16 -- Create the new rule

Once we've created the new rule, we'll land back on the Detection rules management page. We can click on Custom rules to view our new rule:

Screenshot of Elastic Stack Security app Detection view with custom rules
Figure 8.17 -- View custom rules

Clicking on the rule name will open the rule and we can make a final check, make any necessary changes, and activate (or deactivate) it when you are ready to generate events with it:

Screenshot of overview of Elastic Stack Security app Detection rule
Figure 8.18 -- Reviewing the new rule

As a real-world example, you notice in the preceding figure that the destination domain isn't right. I can click on the Edit rule settings button and adjust the domain to include the .com Top-Level Domain (TLD):

Screenshot of rule execution in Elastic Stack Security app
Figure 8.19 -- Packtpub network rule execution

After adjusting my mistake, I can see an execution of the rule and we can even see the severity override we did for the HTTP connection.

We created a custom query rule with KQL; next we'll explore the machine learning rule.

This was last published in November 2021

Dig Deeper on Threat detection and response