Operational FAQ
Administration
Can an existing sandfly-setup folder be moved to a different directory?
- Yes, this can be done without the need to reinstall the software. To perform the move, the basics are to first deactivate all schedules, ensure that no Trickle Schedules are active and that all Task Queues are empty; once empty, shut down Sandfly with the
shutdown_sandfly.shscript, then move the entiresandfly-setupfolder to its new location, and finally restart Sandfly via the appropriate node or server start script.
Is there a way to perform operations in bulk?
- Yes, most of the functionality found in the User Interface (UI) is also exposed via the Sandfly Server's REST API (https://docs.sandflysecurity.com/docs/sandfly-api). The Sandfly setup package includes example API scripts (found under
sandfly-setup/api_examples/) for many common actions which can be used as a working reference or extended to meet your custom needs. Please see the public API Reference (https://api.sandflysecurity.com/) for a complete list of the available calls.
Why did some anti-virus software report that the Sandfly package contains the Sysrv Linux trojan?
- This is a false positive in some anti-virus product heuristic detections. Sandfly uses the non-malicious Cloudflare SSL utility ( https://github.com/cloudflare/cfssl🡵 ), built from source by Sandfly, to generate SSL certificates as part of the Sandfly installation. This false positive has been separately reported as an open issue in the repository ( https://github.com/cloudflare/cfssl/issues/1400🡵 ). Please contact Sandfly Support🡵 should you have any concerns.
How to change the hostname or IP address of Sandfly?
- To change the hostname or IP address of a Sandfly installation, follow these basic steps:
- Stop all active schedules and ensure that the Task Queue is empty.
- Shut down Sandfly (both the Server and all Nodes) via the
shutdown_sandfly.shscript. - As a general recommendation, make a backup of Sandfly files before performing changes.
- Update all hostname values in every node's config file (
config.node.env). - If a custom SSL certificate was in use, update it as well. See: https://docs.sandflysecurity.com/docs/installing-a-custom-ssl-certificate
- Restart the Sandfly Server and all Nodes via the associated start scripts.
- In the sidebar menu, select Settings > Application settings > General. In the Server and data retention card, open its edit form, update Server hostname, and then save your changes.
- If a Let's Encrypt certificate was in use, open the edit form of the Server HTTPS certificate (TLS) card on the same page, set TLS mode to ACME (Let's Encrypt), and then save your changes.
- Finally, re-activate all schedules that were deactivated in the first step.
For details or more information on any individual step, please refer to our official documentation: https://docs.sandflysecurity.com/
Credentials
Can more than one credential be attached to a given set of assets?
- At present, it is not possible to assign more than one credential to a host. This feature has been requested in the past. The complicating factor is that we do not want to activate any kind of auto banning that thinks that Sandfly is brute forcing. If we try 2-3 attempts, that may be OK, but if we try more than that and you use something like fail2ban, then our service could become blocked.
Docker
How to get a complete log from a Sandfly docker container?
- From a shell on either a Sandfly server or node host, use the following commands:
-
docker ps # To obtain the id of the target container docker logs #CONTAINER_ID# &> /tmp/sandfly.log
Can Sandfly scan inside containers?
- Currently, Sandfly can scan the host / infrastructure that runs the containers and from there have full visibility into all of the containers running on that host. Our focus and testing have been on the underlying infrastructure that is running the containers, not container-at-a-time.
- If any container provider runs the underlying infrastructure for you and you do not have SSH access to it, then Sandfly will not be able to connect and scan those containers. However, if those containers are running an ssh server, which is atypical but not unheard of, then Sandfly should be able to connect and scan the inside of the container that way.
Can I clean up inside /var/lib/docker/?
- Sandfly does not provide its own method for cleaning up
/var/lib/docker/due to the risk of data deletion. However, Docker provides the commanddocker system prune(official Docker documentation at: https://docs.docker.com/reference/cli/docker/system/prune/)🡵 that can be used to remove unused data, including volumes with the added--volumesflag, that have accumulated on your Sandfly server and / or nodes. -
IMPORTANT: Volume pruning can delete your database, so be very careful!
If the
sandfly-postgresdocker container is NOT running, volume pruning will delete your database! Also, if non-Sandfly docker containers are or were on a target host, this process will affect every non-running container unless filters are applied. - Before using the command, we first recommend making a backup of the data. Next, open a shell with root privileges on either a Sandfly server or node host. If pruning the Server, check that the container named
sandfly-postgresis running: -
docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 4ff3b716aadf quay.io/sandfly/sandfly:6.0.1 "/opt/sandfly/start_…" 10 days ago Up 10 days 0.0.0.0:80->8000/tcp, :::80->8000/tcp, 0.0.0.0:443->8443/tcp, :::443->8443/tcp sandfly-server 1cb5d910cbbc postgres:18.7 "docker-entrypoint.s…" 10 days ago Up 10 days 5432/tcp sandfly-postgres - Finally, run the prune command, which will remove all unused containers, networks, images (both dangling and unreferenced), and optionally, volumes:
-
docker system prune --all # Optionally add '--volumes' to prune volumes
Protected Hosts
How to update the target address of an existing protected host?
- Add a host with the new target address. Provided that host identifying data has not also changed, Sandfly will update the existing host's target address when it notices that it has the same host ID.
Integrations
- The Settings > Integrations > Third party tab within the Sandfly UI lists all integrations and apps.
How can Sandfly be integrated with Splunk?
- Sandfly offers free Splunk applications in the Splunkbase ( https://splunkbase.splunk.com/apps?author=sandfly🡵 ). Each app listing contains full details and installation information.
Operating Systems (OS)
What commands will Sandfly use on the endpoints?
- When Sandfly first logs in to a host, it uses several native Linux commands, which are run as the associated user, to examine the system in order to determine what that host is. Below is a list of those native commands; full paths are used where possible:
- cat
- chmod
- echo
- grep
- head
- id (additionally run via sudo)
- ls
- mkdir
- pwd
- rm
- sh
- tail
- tr
- uname
- Outside of the native Linux commands, a Sandfly binary is pushed to a randomly named directory in the home directory of the associated user. Its default name is
sandfly; however, that binary name can be changed to one or more alternate names within the server's configuration. The directory will be in the format of<timestamp>.<random_chars>and the entire directory is removed after the scan finishes.
Which Linux distributions can Sandfly protect?
- Sandfly secures the widest range of Linux systems of any product on the market and protects virtually all distributions of Linux.
- This spans, for example, systems that are 10+ years old up to modern distributions at cloud providers. Sandfly works with Linux versions running Intel, AMD, Arm, MIPS and IBM Power CPUs without any special modifications. We also support embedded systems with MIPS or ARM processors such as Raspberry Pi. Basically, because of its agentless architecture, Sandfly only requires that your Linux hosts be running SSH.
- Sandfly has been tested against the following Linux distributions, but will work on many more:
- Alma
- Arch
- CentOS
- CoreOS
- Debian
- Fedora
- Oracle
- RedHat
- Rocky
- Suse
- Ubuntu
- Amazon Linux Images
- DigitalOcean Linux Images
- Microsoft Azure Images
- Raspberry Pi and other embedded systems
- Customized Distributions
What is the oldest OS or kernel that Sandfly can scan?
- Sandfly is not tied to any specific OS distribution, but to kernel age. Beginning with Sandfly version 5.8.0, kernel version 3.2 and higher on the x86 architecture is supported. Other architectures are likely to require newer kernel versions.
Can Sandfly scan Mac OS?
- The Mac OS uses an unrelated kernel, which started life as the Mach kernel as a university research project building on an old BSD Unix kernel, and evolved into the XNU kernel for NeXT, which Apple eventually acquired and used as the basis for Mac OS X. Also, the command line userspace in Mac OS is from the BSD world, not the GNU userspace of Linux.
- Which is all a long way of saying that no, Sandfly cannot scan Mac OS, because despite the similarities of the command line interface, behind the scenes Mac OS is an entirely different beast whose kernel does not share even a single line of code with Linux. And the way Mac OS has evolved, it is even further from how the other non-Linux commercial Unixes do things in a lot of areas.
What TLS ciphers does the server support?
- By default, the Sandfly Server restricts TLS connections to TLS 1.3 cipher suites and a small handful of TLS 1.2 ciphers that SSL Labs🡵 does not flag as potentially weak:
- TLS_AKE_WITH_AES_128_GCM_SHA256 // TLS 1.3
- TLS_AKE_WITH_AES_256_GCM_SHA384 // TLS 1.3
- TLS_AKE_WITH_CHACHA20_POLY1305_SHA256 // TLS 1.3
- TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 // TLS 1.2
- TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 // TLS 1.2
- TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 // TLS 1.2
- Please contact Sandfly Support🡵 if you need to disable the strict TLS policy.
Sandflies
What caused a "Sandfly halted with a SIGKILL signal" error?
- SIGKILL is consistent with what the Linux OOM killer would use to terminate processes. Or there could be a malicious actor on the host.
- If it was the Linux OOM killer, there should be an entry in the dmesg or kern logs. E.g., try
dmesg | grep -i 'out of memory'and see if it mentions anything about killing a Sandfly-associated process. Otherwise, investigate that error as a potential security incident.
How do you change the timeout period of a Sandfly?
- Complete the following steps to deactivate the original Sandfly and in its place use a customized version for future scheduled scans:
- Locate and open the Details page of the sandfly that you would like to customize.
- In the upper right corner, click the Deactivate button if the sandfly is active.
- In the upper right corner, click the Clone button, which opens the custom sandfly editing form.
- In the field where the JSON is displayed, look for the
namekey near the top. The copy is already named<original name>_copy; change it to a unique, descriptive name for your needs, for example<original name>_10mTimeout. - In the same JSON field, look for the
activekey and change its value fromfalsetotrue. Thecustomkey is already set totrue. - In the same JSON field, look for the
max_timeoutkey and change the value to the desired duration. - Save your changes. Sandfly creates a new custom sandfly.
- Adjust any schedules, whitelists, and/or drift profiles as needed to support the custom sandfly.
What happens to custom sandflies as we update and improve the product?
- In the past, we have auto-updated custom sandfly modules to new formats without customers needing to do anything. It is very likely we will continue this tradition for the foreseeable future.
Why is a sandfly alerting on another security product?
Sandfly is expertly crafted to detect unusual or suspicious activity on Linux hosts. Unfortunately, this sometimes includes noticing the well-meaning activities of other security products.
While our sandfly rules have the capability to ignore known / specific cases, we intentionally do not exclude them by default as malicious activity could mimic their environment, thereby bypassing our protection.
In these situations, we first recommend that you thoroughly review any such alerts. If you accept the other security product's activity, use one of these two methods to manage future alerts:
- Use the Whitelisting feature to convert alerts into passes for future results across whichever systems that you determine. For more accurate checks, include the file's hash in the rule, as filenames with paths alone can potentially be more easily replaced.
- Use the Drift Detection feature to lock down the binary's hash in an associated profile.
Please note that when hashes are used in rules, any change, be it due to a patch, file update, replacement by malicious activity, etc., will probably change the value of the hash. This will cause alerts to reappear, so occasional re-review and rule management will be necessary when using hashes.
Why is a sandfly alerting on a certain installed package or application?
Sandfly is tested on many default distribution installs with no false alarms. However, each customer deployment is different and it may be necessary to whitelist a small number of false alarms depending on your environment.
Sensitive Data
Is sensitive data encrypted in transport (e.g., system-to-client)?
- We encrypt all interactions from the UI to the server. We also encrypt all data to/from the systems being investigated and also that data sent back to the server.
Is sensitive data encrypted in storage (e.g., disk encryption, at-rest)?
- We encrypt all credentials we store in the server at all times. However, results, server configurations, etc. are not encrypted in the database.
Updated 1 day ago