If you are using Sandfly, especially in a production environment, it is important to take precautions to ensure that your data and configurations are not lost. While results data comes and goes relatively quickly, other application data such as custom sandflies, schedules, configured hosts, accounts, credentials, and other irreplaceable data is important to maintain through regular backups.

In this guide you will learn how to backup and restore Sandfly Nodes and the Server at the file system level.

Directory Structure Overview

The general Sandfly system essentially consists of two areas / directory structures:

Sandfly Application
- The root directory is "sandfly-setup", which mainly contains scripts and configuration text files that are needed for the application
- There is no default or required path for the installation.
  - Consult with the System Administrator who initially installed / will be (re-)installing Sandfly for the location defined by your environment.
  - Alternatively, the location can likely be found via this command:
    - find / -name sandfly-setup
  - With the exception of the directory mentioned below, other files within Sandfly's root come directly from the downloaded package, thus it should not be necessary to backup those files.
- Important directory:
  - <SANDFLY_ROOT_DIR>/sandfly-setup/setup/setup_data/
    - This directory, along with its sub-folders, contains all of the file system level configuration text files that are unique to your installation.
    - The total size of this recursive path should be less than 1M.
    - Files within this directory can be safely backed up at any time and does not require the Sandfly application to be stopped.
Docker Service
- Sandfly files that are contained within the Docker service.
- The default docker root path is: /var/lib/docker/
  - This path can be different, depending on your environment.
  - Use the following command to find the root path for Docker:
    - docker info --format='{{.DockerRootDir}}'
- Important directories:
  - <DOCKER_ROOT_DIR>/volumes/sandfly-pg14-db-vol
    - In general, the "volumes" directory contains persistent data used by Docker containers.
    - Specifically the "sandfly-pg14-db-vol" directory contains the entire Postgres database used directly by Sandfly.
      - The full path to the folder of the database can be systematically generated with this command:
        docker inspect sandfly-pg14-db-vol --format='{{.Mountpoint}}'
    - All other directories under volumes are automatically (re-)generated and thus do not need to be backed up.
    - A "human readable" size of the database files can be determined with this command:
      du -h --total `docker inspect sandfly-pg14-db-vol --format='{{.Mountpoint}}'` | tail -1
    - A database backup at the volumes level requires that the Sandfly application is first stopped, otherwise data corruption can occur from unfinished transactions.
  - <DOCKER_ROOT_DIR>/containers/
    - In general, the "containers" directory holds files for the active Docker containers.
    - The log files that are located under this base path can be backed up should there be a need to do so for any business or technical reason. For the operation of the application itself, a backup is not required at all as these files are not persistent.
      - A full path list of all available container log files can be systematically generated by this command:
        ls -1 `docker info --format='{{.DockerRootDir}}'`/containers/*/*-json.log*
    - A "human readable" size of the optional Docker log files can be determined with this command:
      du -h --total `docker info --format='{{.DockerRootDir}}'`/containers/_/_-json.log\* | tail -1
    - These logs files can be safely backed up at any time and does not require the Sandfly application to be stopped. However, as some logs may be very active, truncated logs can occur with a running server.

Backing Up

Backup details are broken out below into sections for the server and nodes.

Sandfly Server

For a typical server installation the configuration text files should rarely change, if at all, with the exception when using a custom SSL certificate within the server_ssl_cert directory. On the other hand, the database contains regularly changing data … depending on the size and scope of your environment of course.

File System Approach Overview

The goal of this backup approach is to make a copy of all of the Sandfly configuration text files, the entire database (at the Docker volume level), and optionally the log files of the Docker containers, as they exist on the file system of the server at the moment of the backup action. If there is a large quantity of Sandfly results, the backup size can be very large.

As a part of using this approach, the Sandfly application will need to be stopped during the process. This is to allow the database to be in an inactive state in order to avoid data corruption.

If you require a non-service disrupting backup of the Sandfly database, that approach is possible via supported Postgres SQL dumping methods. However, those steps are currently not covered in this guide.

File System Steps

Perform all of these steps on your Sandfly server:

Deactivate all Active Schedules via the User Interface (UI).
1. Quicklink: Deactivating and Deleting Schedule
2. WARNING: This step is important as any schedules that were left as enabled will be backed up as such. Thus, after a database restore, those enabled schedules will run immediately once the server resumes full operation.
Wait for the Task Queue to empty.
1. In the Sandfly User Interface (UI), check the Task Queue by clicking on its button in the Top Bar or via the menu at Scanning > Task Queue and make sure the queue is at 0 (zero), as indicated by the Total Tasks value.
Shutdown the Sandfly containers on the Server host in a safe manner.
1. <SANDFLY_ROOT_DIR>/sandfly-setup/start_scripts/shutdown_sandfly.sh
Ensure that the following items are copied (recursively if a directory) by the backup system of your choice:
1. <SANDFLY_ROOT_DIR>/sandfly-setup/setup/setup_data/
2. <DOCKER_ROOT_DIR>/volumes/sandfly-pg14-db-vol/
3. <DOCKER_ROOT_DIR>/containers/_/_-json.log* (Optional)
Start the Sandfly server.
1. <SANDFLY_ROOT_DIR>/sandfly-setup/start_scripts/start_sandfly.sh
Verify that the sandfly-server and sandfly-postgres containers are running with the following command:
1. docker ps
Activate all Schedules that were deactivated in Step 1.

Sandfly Node

The goal of this backup is to make a copy of all of the Sandfly configuration text files, and optionally the log files of the Docker container(s), as they exist on the file system of a node at the moment of the backup action.

For a typical node installation the configuration text files should rarely change, if at all. This file system level backup can be performed at any time and does not require the Sandfly application to be stopped.

File System Level Steps

Perform all of these steps on each of your Sandfly nodes:

Ensure that the following items are copied (recursively if a directory) by the backup system of your choice:
1. <SANDFLY_ROOT_DIR>/sandfly-setup/setup/setup_data/
2. <DOCKER_ROOT_DIR>/containers/_/_-json.log* (Optional)

Restoring

Restoration details are broken out below into sections for the server and nodes.

Sandfly Server

Data can be recovered completely or partially, under certain circumstances. Each case will influence options that are a part of the restoration process.

File System Approach Overview

The goal of this restore approach is to rebuild server files solely at the file system level. This means that the database directory must be changed in its entirety within the Docker volume folder and not at a database table level. In addition, the Sandfly application will need to be stopped during the process.

If you need SQL level or partial restorations of the Sandfly database, that approach is possible via a combination of supported Sandfly and Postgres methods. However, those steps are not covered in this guide.

Based on your recovery situation, this process considers and allows for the restoration of either the entire set of files (such as may be needed for a complete server rebuild) or selective, non-database files of (such as from an accidental text file deletion or corruption). To reiterate, do not partially restore any files in the database volume. Lastly, the Docker container logs should never be restored into their original location.

File System Steps

Perform all of these steps on your Sandfly server:

If the Sandfly server is NOT running, skip to Step 4.
Deactivate all Active Schedules via the User Interface (UI).
1. Quicklink: Deactivating and Deleting Schedule
Wait for the Task Queue to empty.
1. In the Sandfly User Interface (UI), check the Task Queue by clicking on its button in the Top Bar or via the menu at Scanning > Task Queue and make sure the task queue is at 0 (zero), as indicated by the Total Tasks value.
Shutdown the Sandfly containers on the Server host in a safe manner.
1. <SANDFLY_ROOT_DIR>/sandfly-setup/start_scripts/shutdown_sandfly.sh
  1. Any "No such container" errors can be safely ignored.
Copy files from your backup system.
1. NOTE: If any default directories under <DOCKER_ROOT_DIR> are missing or corrupt, they can be recreated by simply restarting your docker service before attempting to restore your backup files.
2. If restoring only specific text files within "setup_data" directory, copy those files starting at the following directory:
  1. <SANDFLY_ROOT_DIR>/sandfly-setup/setup/setup_data/
3. If restoring the entire server backup, ensure that all files within the following two directories are copied recursively:
  1. <SANDFLY_ROOT_DIR>/sandfly-setup/setup/setup_data/
  2. <DOCKER_ROOT_DIR>/volumes/sandfly-pg14-db-vol/
    1. The "containers" path, which holds the container log files, should never be restored into its original path. If log file(s) need to be restored for any reason, restore them into an alternate location.
Start the Sandfly Server
1. <SANDFLY_ROOT_DIR>/sandfly-setup/start_scripts/start_sandfly.sh
Verify that the sandfly-server and sandfly-postgres containers are running with the following command:
1. docker ps
If any or all of your Nodes also need to be restored, complete the entire node restore process before proceeding to the next step.
Resume all Schedules that were deactivated in Step 2 (if applicable).
1. NOTE: If you have many schedules to enable, especially those that run on the same time cycles and/or share the same set of hosts, consider enabling schedules over a period of time.

Sandfly Nodes

Based on your recovery situation, this process considers and allows for the restoration of either the entire set of files (such as may be needed for a node rebuild) or selective files (such as from an accidental file deletion or corruption).

As the "config.node.json" file directly affects the running configuration of the node, all containers on that host will need to be stopped. If this is your only node, then Sandfly scanning functionality will be unavailable until the node is back in service. However, should you have additional nodes that are operational, cover the same Queue, and the server is operational, then scanning should still be able to continue. On a final note, the process below also assumes that only one node will be restored at a time, unless the server is inactive.

Finally, the Docker container logs should never be restored into their original location.

File System Steps

Perform all of these steps on each of your Sandfly nodes as needed:

If the Sandfly Server is stopped for any reason, OR if there is more than one operational Node for the same Queue, skip to Step 4.
Deactivate all Active Schedules via the User Interface (UI).
1. Quicklink: Deactivating and Deleting Schedule
Wait for the Task Queue to empty.
1. In the Sandfly User Interface (UI), check the Task Queue by clicking on its button in the Top Bar or via the menu at Scanning > Task Queue and make sure the task queue is at 0 (zero), as indicated by the Total Tasks value.
Stop all of the Sandfly containers on the Node host.
1. docker stop $(docker ps -a -q)
Select either specific files (for a targeted restore) or the following directory (for a full, recursive restore) from your backup system and copy those files on to the node system at this location:
1. <SANDFLY_ROOT_DIR>/sandfly-setup/setup/setup_data/
Start the Sandfly node.
1. <SANDFLY_ROOT_DIR>/sandfly-setup/start_scripts/start_node.sh
  1. Repeat Step 6 for each node instance that is expected to run on this host.
Verify that the expected number of sandfly-node container(s) are running.
1. docker ps
Repeat Steps 4-6 for each host that is functioning as a Sandfly Node.
Activate all Schedules that were deactivated in Step 2 (if applicable).

This concludes the Backup and Restore Guide via the file system method. Other data backup methods, such as via SQL or VM snapshots are also possible. The important factor is to have Sandfly in an appropriate state where you are able to backup the data free from errors that could otherwise be caused by incomplete transactions.