Sandfly Security

Sandfly Security Documentation

Welcome to the Sandfly Security documentation hub. Sandfly is an agentless compromise and intrusion detection system for Linux.

Sandfly automates security investigation and forensic evidence collection on Linux. Sandfly constantly searches for intruders 24 hours a day on your network without needing to load any agents on your endpoints.

Get Started

Node Install

Sandfly Node Installation

Sandfly Node Install

Sandfly nodes allow the system to connect to remote hosts to do our agentless investigations and analysis. You need at least one node running at all times, but it is recommended you start up multiple node containers under Docker for redundancy and performance reasons.

You must run server and nodes on separate systems for security reasons.

It is fine to run multiple node containers on the same host if you have the RAM to do so. However, we do not recommend you run nodes on the same instance as the server. The nodes store secret keys that protect SSH credentials stored in the server from compromise. If you have both the server and node loaded on the same system, a compromise can put your SSH credentials at risk of theft. We recommend you always run nodes on their own instances for this reason.

Download Setup Scripts

Install the Sandfly setup scripts from the Github repository. Clone them onto your system. You will need to have git installed on your system with one of the following commands on CentOS or Ubuntu:

sudo yum -y install git

OR

sudo apt install git

Now use git to download the install scripts:

git clone https://github.com/sandflysecurity/sandfly-setup.git

You will see a directory called sandfly-setup. We will do our work inside this directory for the rest of this document.

Install Docker

Sandfly uses Docker to run its containers. Please install Docker on your host. It is important to use the latest version of Docker. Ubuntu and Centos repositories contain very old versions of Docker and are not compatible with Sandfly. Please use the install scripts below to install the latest versions:

Ubuntu and CentOS Repositories Are Too Old for Docker

Some Linux distributions (such a CentOS) contain old versions of Docker and are not compatible with Sandfly. Please install using the newest versions of Docker from the scripts provided.

Centos Docker Install

~/sandfly-setup/setup/install_docker_centos7.sh

Ubuntu 17 and Older Docker Install

~/sandfly-setup/setup/install_docker_ubuntu.sh

Ubuntu 18 and Newer Docker Install

~/sandfly-setup/setup/install_docker_ubuntu18.sh

Debian 9 and Newer Docker Install

~/sandfly-setup/setup/install_docker_debian.sh

Make sure the Docker daemon starts automatically or you can start it manually on Linux with the following command:

service docker start

Login With Your Docker ID

Docker Login Required

Before you can access the Sandfly Docker repository, you will need a valid Docker account and this account will need to be added to read the repository by Sandfly.

Please setup a Docker account and let us know what the Docker ID is so we can add it so you can perform the install.

Login with your Docker ID to push and pull images from Docker Hub. If you don't have a Docker ID, head over to hub.docker.com to create one.

To login with your Docker ID, you will need to do the following:

docker login
Username: exampleuser
Password:
Login Succeeded

Copy Over Keys to Node

We now need to load the passwords, cryptographic keys, and hostnames generated during the server install into the node. This will require some cut and pasting, or you can use scp to copy the files over all at once.

You will want to open two terminal windows. One will need to be connected to the SERVER, and the other to the NODE.

Go to the sandfly-setup/setup_data directory on the SERVER:

ON SERVER:

cd ~/sandfly-setup/setup/setup_data

Go to the sandfly-setup/setup_data directory on the NODE:

ON NODE:

cd ~/sandfly-setup/setup/setup_data

Copy Over Files from the Server to the Node

You can use scp, or cut and paste between screens. But the files below must all be copied from the SERVER to the NODE.

cacert.b64
node_cert.b64
node_key.b64
node.sec.asc.b64
rabbit.server.hostname.txt
rabbit.node.password.txt
api.server.hostname.txt
api.node.password.txt

DELETE THE NODE SECRET KEY

The server generated a PGP public/private key pair for the scanning nodes. However, we don't want the server to have the private key.

Once you copied node.sec.asc.b64 onto the node, you can go into the SERVER setup_data directory and delete this file. The server only needs the node.pub.asc.b64.

Secure delete:

ON SERVER:

shred ./node.sec.asc.b64

Or standard delete:

ON SERVER:

rm ./node.sec.asc.b64

DELETE THE SECRET KEY

You must delete the secret key from the SERVER to ensure full security of your SSH credentials with Sandfly.

Start One Node Container

Once you deleted the secret key from the server, then you can start the node. Docker images will be pulled over and the node will start if the keys above were copied over correctly.

Note that if you used a signed SSL certificate during the install you run the script below.

cd ~/sandfly-setup/start_scripts
./start_node.sh

If you did not sign the SSL key during the server install, you must run the following script to tell the node to ignore the signature status of the server.

cd ~/sandfly-setup/start_scripts
./start_node_unsigned_cert.sh

Run the Correct Node Start Script!

You must run the start_node_unsigned_cert.sh script to tell the nodes to ignore the unsigned certificate if you did not sign the certificate! If you don't do this, then the nodes will refuse to send data back to the server with the unsigned certificate.

Start Multiple Node Containers

You can start multiple node containers on the same system to get more performance and redundancy by simply running the start_node.sh script repeatedly (or start_node_unsigned_cert.sh). Make sure your host instance has the RAM to run multiple node containers before doing this.

We Recommend Running Multiple Containers

We recommend you run multiple node containers. You can run multiple containers on a single host instance or on individual hosts. Running multiple containers provides much higher performance and redundancy if a container exits unexpectedly.

Each node container runs 500 scanning threads. So for each node container you add onto the system you expand scanning capacity by 500 threads.

Running 5 nodes for instance gives you 2500 scanning threads. This means that you can scan 2500 hosts concurrently. It also means that if one container should die unexpectedly, you will still have capacity for scanning to continue uninterrupted.

root@example:~/sandfly-setup/start_scripts# ./start_node.sh
0106c87dbfd304b3f6fff847702a41f603eb5e625c7b6194ba5fd30019533421

root@example:~/sandfly-setup/start_scripts# ./start_node.sh
9ecc25cdaae72589d4792b01989ab73001bcf400da05cfd436a54e9defc38be9

root@example:~/sandfly-setup/start_scripts# ./start_node.sh
a8c3b80228c47a7feabf0dfbee89cbd6a2d5abbe80ec7b2a61fc86ed246bfbd7

You can run the command below to see all the nodes containers running.

docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS               NAMES
a8c3b80228c4        sandfly/sandfly-node:latest   "/usr/local/sandfly/…"   3 seconds ago       Up 1 second                             gallant_neumann
9ecc25cdaae7        sandfly/sandfly-node:latest   "/usr/local/sandfly/…"   4 seconds ago       Up 3 seconds                            confident_knuth
0106c87dbfd3        sandfly/sandfly-node:latest   "/usr/local/sandfly/…"   6 seconds ago       Up 4 seconds                            ecstatic_shockley
fb25ff348c30        sandfly/sandfly-node:latest   "/usr/local/sandfly/…"   7 seconds ago       Up 6 seconds                            elegant_lovelace

Node Container RAM and CPU

Make sure your host instance for the node containers has enough RAM before running many containers and a couple CPUs to make sure there are no performance issues.

A 2GB instance can run 4 containers comfortably. A 4GB instance can run around 10 node containers or perhaps more.

If you want to run many node containers on a single instance you will need to scale up RAM and CPU accordingly.

If you want, you can view the log of the node to make sure it is connected properly. Do this by finding out what the Docker log is called for output after you run the above.

Use the docker name for the container to find what the unique log name is for that container instance:

docker inspect gallant_neumann | grep log
        "LogPath": "/var/lib/docker/containers/a8c3b80228c47a7feabf0dfbee89cbd6a2d5abbe80ec7b2a61fc86ed246bfbd7/a8c3b80228c47a7feabf0dfbee89cbd6a2d5abbe80ec7b2a61fc86ed246bfbd7-json.log",
        
        

With the above, you can then view the log and watch information scroll by:

tail -f /var/lib/docker/containers/a8c3b80228c47a7feabf0dfbee89cbd6a2d5abbe80ec7b2a61fc86ed246bfbd7/a8c3b80228c47a7feabf0dfbee89cbd6a2d5abbe80ec7b2a61fc86ed246bfbd7-json.log

{"log":"Adding certificate for server.\n","stream":"stdout","time":"2018-05-31T03:52:58.445303074Z"}
{"log":"Adding key for server.\n","stream":"stdout","time":"2018-05-31T03:52:58.450253392Z"}
{"log":"Changing rabbit and API passwords in conf files to supplied value.\n","stream":"stdout","time":"2018-05-31T03:52:58.467539835Z"}
{"log":"Changing server URL to supplied value.\n","stream":"stdout","time":"2018-05-31T03:52:58.482353609Z"}
{"log":"Starting Celery as lone node\n","stream":"stdout","time":"2018-05-31T03:52:58.484690779Z"}
{"log":"[2018-05-31 03:53:02,275: INFO/MainProcess] Connected to amqp://node:**@example.sandflysecurity.com:5673//\n","stream":"stderr","time":"2018-05-31T03:53:02.279679495Z"}
{"log":"[2018-05-31 03:53:02,380: INFO/MainProcess] mingle: searching for neighbors\n","stream":"stderr","time":"2018-05-31T03:53:02.381229883Z"}
{"log":"[2018-05-31 03:53:03,486: INFO/MainProcess] mingle: sync with 2 nodes\n","stream":"stderr","time":"2018-05-31T03:53:03.487267081Z"}
{"log":"[2018-05-31 03:53:03,489: INFO/MainProcess] mingle: sync complete\n","stream":"stderr","time":"2018-05-31T03:53:03.489423504Z"}
{"log":"[2018-05-31 03:53:03,529: INFO/MainProcess] node.2018-05-31T03.52.58@a8c3b80228c4 ready.\n","stream":"stderr","time":"2018-05-31T03:53:03.530126446Z"}

Once this is done, you can go back to the server UI and initiate a scan and you will see the node messages scroll by (very fast).

Node Install


Sandfly Node Installation

Suggested Edits are limited on API Reference Pages

You can only suggest edits to Markdown body content, but not to the API spec.