Introduction

Most multi-tier applications benefit from scaling to allow them to function more effectively and reliably. Sandfly is no exception. An overloaded Node will eventually prevent scans from completing or an under-scoped Server could cause degraded performance or complete loss of service. We encourage Sandfly users to scope and build out their Sandfly deployments sufficiently to accommodate your operating environment and technical requirements. The information provided below will aid you in accomplishing this.

Scaling Goals

In general, the overall goal is to avoid loss of service or performance degradation. While Sandfly is designed to be robust, inadequate system resources could lead to situations such as overwhelming the node's ability to clear its task queue expediently, insufficient CPU and RAM on the server to process or store results data, or simply having the container partition becoming full due to the quantity of results in relation to the data retention period. These and other cases can be minimized by architectural choices.

What that means from a quantitative perspective is dependent on various factors, which predominantly consist of:

• How many active hosts are available to scan?
  • Each active host is essentially a multiplier of processing and data storage.
• How many enabled Schedules exist and what percent of hosts are included?
  • The more scans that are run, the more data that will be produced, processed, and stored.
• How many Sandflies run per Schedule on average and of which Types?
  • Some Sandflies take longer to run and/or produce more data than others.
  • See the Sandfly Types documentation for details about the different types.
• Are scanned hosts on the same network segment, shared storage, and/or physical hardware?
  • Scans simultaneously occurring on endpoints with shared resources, especially filesystem-based, will increase resource use on the common component.
• Are scanned hosts physically located together or are they geographically and/or service provider distributed such as in multiple data centers?
  • Named Queues benefit these environments, but require additional nodes.
  • See the Named Queues documentation for additional details.

The answers to those questions should be used in the scaling calculations and configuration decisions.

Server

Sandfly uses a single Server, which supplies the web interface, REST API, and database.

An under-powered server could have database timeout issues. Or, if the User Interface is taking a long time to load data, then there could be too few resources and be in need of an upgrade of the RAM and/or adding additional CPU cores.

Server Resource Requirements

The Sandfly stack consists of two containers, one for the API and web interface and one for the local database.

The absolute minimum Server system resource requirements:

• 4GB of RAM
• 2 dedicated CPU cores
• 20 GB of disk space (includes the host Operating System (OS))
  • SSD drives are recommended for better application performance.

Each system resource will need to be scaled up independently, where and when it is appropriate for your environment.

Server Scaling

The main focus when scaling the Server is to try to prevent the database from becoming overloaded or going into a blocking state. In most cases, scaling will be done at the host level; therefore, running the Server in a Virtual Machine (VM) will allow for easy system resource changes as the needs arise or grow.

Server scaling areas to take into account:

• For database performance
  • Database performance for interactive queries (e.g., user actions in the UI) is improved when the database server can keep indexes in RAM. The Sandfly start scripts configure Postgres to use a percentage of the server's available RAM: increasing RAM in the Server VM will make more RAM available for Postgres to cache indexes in memory.
  • When the disk subsystem is fast enough, the bottleneck during result ingestion is CPU capacity. If the result ingestion queue is high and you see Postgres processes using 100% CPU (and if the ingestion queue isn't being processed at a rate you are satisfied with, e.g., the new scheduled scans are happening before the previous scans have been ingested), additional (or faster) CPU cores improve the ingestion rate.
• For disk space
  • Partition sizing applies to the location that contains the Docker data.
    • By default that path is /var/lib/docker/ for Docker. For Podman, it is /var/lib/containers/ (rootful) or ~/.local/share/containers/ (rootless).
  • The vast majority of the database size will come from scan results data and the indexes on that data.
  • A single scan pair (Sandfly + os_identify) generates approximately 2k of data.
  • The aggregated size varies greatly, depending on several factors:
    • The nature of the workload of the scanned system:
      • More processes running leads to more results
      • Higher process churn (different PIDs running each time Sandfly scans the target) reduces deduplication of data, and therefore more result data in the Sandfly database
      • More complex processes (high number of network sockets and open file descriptors) lead to larger result sizes
    • Scan schedules, which include:
      • The run frequency of enabled schedules
      • The percentage, types, and quantity of selected sandflies
      • The quantity of included, active hosts per scan
    • Data retention period:
      • Data retention - between 1 and 31 days, depending on the license and settings.
      • Pass data retention - a separate period for pass results, also limited by the license.
      • Results are deleted when their retention period ends.
  • Should longer periods of data retention be required, we recommend that the data is replicated into an external data store via any of our available integration methods.

Server Schedules

Sandfly Schedules are the largest source of data generation as they drive automated scans.

By default, scan schedules spread the scans over the scheduled interval (trickle mode) to reduce the load on target systems that share underlying resources (such as the same VM host or SAN storage), but schedules can be set to immediate mode to trigger all scans immediately at the schedule time. Manual scans initiated by users (or via API) always run immediately.

Because of those points, it is important to plan your Schedules with sufficient forethought to minimize the chances of scans not completing and to not overload the Nodes.

Nodes

Sandfly Nodes allow the application to connect to Linux hosts over SSH to perform agentless scans and report the results back to the Server. Each Node container runs 500 scanning threads by default.

Node Resource Requirements

The scanning nodes consist of one or more Docker or Podman containers. Should the host's available memory allow it, you can, and in most cases should, run multiple Node containers on a single system instance.

The minimum Node system resource requirements:

• 4GB of RAM
• 1 dedicated CPU core
• 5 GB of disk space (includes the host Operating System (OS))

That configuration provides the capacity to run up to 3 Node containers simultaneously, depending on the quantity of hosts included in a scheduled scan.

• Each container provides 500 scanning threads by default.
• Provides a total of 1500 scanning threads, which is sufficient for smaller deployments that do not require host-level redundancy.
• After reserving the first GB of memory for the base OS, a ratio of 1 GB of memory per 1 Node container is a reasonable ratio for scaling consideration.

Node Scaling

The main Node scaling goal is to not have the overall rate of scan additions into the task queue overwhelm the ability of the node(s) to clear the queue out. Thus, the ratio of Node threads to scanned hosts does not have to be 1:1.

In fact, there is nothing wrong with having 800 tasks in the queue but only 500 threads of node capacity available; as each task finishes, the next waiting task will be picked up by the next node thread that becomes available.

This is an important factor to consider with the overall architecture, in that it may be gentler on your endpoint hosts and network, especially if they are all running on the same VM servers, shared storage, etc.

Sandfly Nodes can be scaled in 3 different ways, which are largely influenced by your environment and technical requirements:

• Vertical - The most basic form of Node scaling involves adding more memory in order to allow for additional Node containers to run on a single system. Use of only vertical scaling is the least redundant form of scaling; however, for low-impact and/or small deployments it usually is sufficient.
• Horizontal - For reliability and redundancy, we recommend having 2 (or more, as appropriate) Node systems available, preferably on different infrastructures. Should a Node fail for whatever reason, the remaining Node(s) can continue scanning. These nodes can also benefit from the Vertical scaling method. Just keep in mind the points above so as to not over / under-architect.
• Named Queues - This scaling direction is beneficial for distributed, segmented, and/or large environments. Sandfly has the ability to deploy scanning nodes inside multiple isolated networks and communicate to the central server. This configuration allows you to deploy Sandfly at multiple cloud providers, remote offices, internal networks and other layouts while allowing all nodes to be controlled from a central point.
  • Should your environment fall within this last layer of architecture, we recommend the use of Horizontal and Vertical scaling for each Named Queue.

Node Charts

Conceptual Minimal Scaling per Node Host (without container redundancy)

CHART NOTES:

• Not suggested for critical infrastructure; the data is provided mainly for purposes of scaling considerations.
• Does not consider the use of Named Queues.

Starting Conceptual Minimums (provides both container and host-level redundancy)

CHART NOTES:

• Should one of the multiple node hosts fail, the remaining node host(s) need to be able to maintain the full load. The last column shows the scanning threads that remain when one node host is down.
• Does not consider the use of Named Queues.

Measuring

Finally, the use of system monitoring tools that regularly collect system resource data, such as CPU load, memory use, disk I/O, network use, etc., on all of your hosts will aid your ability to scale, fine-tune your configuration, and track down application-affecting issues.

Aside from generally being able to be alerted for maxed-out resources, which will cause problems on any host, the same data can be analyzed when less impacting situations occur, such as Sandfly scan timeouts. Additionally, this data creates baselines that are useful for seeing before and after conditions, such as changes caused by increasing the quantity of sandflies or adding additional Schedules, for a couple of examples.

Conclusion

By applying the above best practices and scaling information, Sandfly can be set up to run reliably in any size of environment.

Should you require further assistance with scaling and/or architectural configurations, please reach out to your Sandfly representative.