Example: Drift Detection

In this example, we will use the drift detection feature of Sandfly to monitor a set of servers that are created from the same base image and should always be configured identically, running the same set of processes, etc.

To begin, we have all of our "webhead" servers in Sandfly. All of them are tagged with a common tag that identifies the base VM image we used to create the servers.

We will use one of these hosts, mw-webprod-node-01, as our "model host" on which to build the profile of expected results.

Step 1: Gather results on which to base the profile

To ensure we have a clean slate when creating our profile, we first delete all of the results on our model host.

1. Go to Results By Host and click on the host.
2. Check the "select all" box at the top of the results table.
3. Click the Delete button in the table header to delete all results for this host.

Now we can gather the results for the profile. Begin a new manual scan of the host by clicking the "Scan Now" button at the top of the Sandfly UI.

In the Create New Scan dialog, select the model host for the profile, mw-webprod-node-01 in this case. Click Next.

To select sandflies, click on Presets and choose Active and Type Recon. All sandfly types can be used in result profiles to support automatic whitelisting, but typically only recon sandflies are useful for drift detection.

Use the "select all" checkbox to select all recon sandflies and click Finish to begin the scan.

NOTE: The sandflies you select will determine which aspects of the system are monitored for drift. The recon_kernel_models sandfly, for example, will allow drift detection to alert on any new kernel module that isn't part of your model host. Some recon sandflies, such as recon_log_list_lastlog and other log list sandflies, may produce undesirable drift alerts if you expect users to be logging in to the set of servers you apply the profile to. When creating result profiles for drift detection, be sure to deselect any recon sandflies in areas you do not want to monitor for drift at this point.

Step 2: Create the result profile

After the scan finishes, the model host has results from which we can create the result profile.

1. Go to Results By Host.
2. Check the select box next to the model host with the results for the profile.
3. Click the Create Profile button in the table header.

Now we provide the details of the profile.

1. Name the profile.
2. Provide a description of the profile that will help remind users what the profile is for.
3. Optionally enter additional user notes. In this case, we indicate which base VM image this set of results reflects.

After clicking Next, we have the opportunity to use this profile and associate it with hosts.

1. Select "Host Drift Detection" to indicate how we would like to use this profile.
2. Enter the appropriate host tag(s) that we would like to apply the drift detection to. In this case, we select our webhead_v20240105 tag that all of our similar hosts are tagged with.
3. Since all of our hosts were created from the same VM template, we know they should have the same version of executables and other attributes should match very closely. We will select the Moderate Hash Match to enforce a stricter set of conditions to identify drift.

After clicking Finish, the profile will be saved.

Step 3 (optional): View the profile

From the Result Profiles sidebar menu option, we can view the list of profiles and click on the new profile that we just created.

From the details page, we can see some information about the profile, and by selecting the Host Coverage tab, we can confirm that this profile is now associated with all of our webhead hosts via the tag we selected.

Drift Results

At this point, the profile is created, applied to the tagged hosts, and all future scans will analyze results based on the result profile. After some time passes, we notice that one of the webhead hosts is beginning to generate alerts:

Let's examine the host's results to see if we have a problem.

The recon_log_list_... results are due to new entries in the system's lastlog. If we don't expect users to ever log on to our production servers, these alerts may be valuable indications that a compromise may have occurred. However, if we do expect different users to log on to the systems, we may wish to exclude these sandflies from the list of recon sandflies we selected in step 2.

Let's look at the process and user alerts, though.

This result shows that the recon_process_list_all sandfly found a process running, named httpd at /usr/bin/httpd, that was not part of the results we created the profile with. This is suspicious: our real web server process that was part of the profile probably ran from /usr/sbin and almost certainly wasn't running with a process.username of root. This may be a malicious process that has been placed on this server that is trying to masquerade as the legitimate web server and should be investigated further.

Additionally, we see a user list alert.

A new user called _httpd exists on this server that was not part of the model host. Again, this is consistent with the unexpected process: it appears an attacker is attempting to establish a presence on the server and masquerade as legitimate httpd activity. However, because we know that /usr/bin/httpd process and the _httpd user are not part of our normal web server system build, Sandfly allows us to immediately identify these drifts from the expected results as suspicious.

Conclusion

The drift detection capability that Sandfly provides is a powerful tool to quickly identify potential threats by alerting on deviations from expected results on a set of similar hosts. Creating drift detection profiles and applying them to hosts is easy: scan a host to gather the set of expected results, create a new profile from the host, and apply that profile to other hosts with a shared tag. Sandfly will automatically examine all future scan results through the lens of the profile and alert on any new unexpected activity on the hosts.

Drift detection is a valuable tool for monitoring embedded devices, fleets of servers created from the same image, or even any single system that should have a consistent set of activity to identify any deviations from a known-good recon result set.