HomeDocumentationAPI Reference
Log In
Documentation
Start typing to search…

Elastic Connector

Elastic Connectors are pre-built software packages that facilitate data syncing from a specific source into Elasticsearch. These connectors are self-managed from your own infrastructure using a GitHub package located at https://github.com/elastic/connectors.

The Sandfly Elastic Connector is included in the source code distribution. There are several steps required to setup and configure the Sandfly Elastic Connector.

  1. Create a Connector in Kibana.
  2. Download the Elastic Connector source code to your own infrastructure.
    • Create the config.yml file in the Connector service source.
    • Run the Connector service on your own infrastructure.
  3. Finish configuring the Connector settings in Kibana.
  4. Modify the index Settings and Mappings for the Sandfly documents.
  5. Run a Full Content Sync to initialize the index.
  6. Schedule an Incremental Sync to continuously add new data to the index.
  7. Install example Sandfly Dashboards and Reports.
📘

NOTE: Connector Creation Process May Vary Depending on Version

Depending on the version of Elasticsearch/Kibana that you are using, the process to create a connector may be slightly different than described here. Elastic Cloud and older versions of Elasticsearch/Kibana may have a slightly different interface and may or may not automatically create an index and API keys, but the overall process is the same.

  1. Create a Connector in Kibana.
  2. Configure and run the Connector source code on your own infrastructure.
  3. Finish the Connector configuration in Kibana.

Create a Connector in Elasticsearch/Kibana

In the Kibana console, navigate to or search for the Connectors section. It may also be called Data / Content Connectors.

  1. Click on New connector.
  2. On the Create a connector page, find the Connector dropdown, click on Choose a data source and select Customized connector.
  3. Enter the Connector name, for example sandfly-connector.
  4. Click Next to begin deployment and configure the Index, API key, and Connector ID.
  5. On the Deployment page, these instructions assume using the Run from source option, however, you may also choose to Run with Docker.
  6. Copy the Connector ID and the API Key which will be used to configure the Connector service.
  7. You should see a message similar to Waiting for your connector to check in

Configure and Run the Connector Service

📘

NOTE: The Connector Service Requires a Linux System in Your Infrastructure

The Elastic Connector service is intended to be run on a Linux system in your own infrastructure. This host must have Python 3.10 or 3.11 in order to run the Connector service source code. For the security of your Sandfly installation, we do not recommend using the Sandfly hosts for this service.

These instructions assume running the Connector service from source. If you prefer to run the Connector service via Docker, see the Elastic documentation here: https://www.elastic.co/docs/reference/search-connectors/es-connectors-run-from-docker

Clone or download the Connector service source code to your Linux system:

git clone https://github.com/elastic/connectors

Create the connectors/config.yml file:

connectors:
- connector_id: <YOUR_CONNECTOR_ID>
  service_type: sandfly
  api_key: <YOUR_API_KEY>
elasticsearch.host: <YOUR_ELASTIC_HOST>
elasticsearch.api_key: <YOUR_API_KEY>
elasticsearch.verify_certs: false
  1. Set the connector_id to your Connector ID copied from above.
  2. Set the service_type to sandfly.
  3. Set the api_key for your connector copied from above.
  4. Set the elasticsearch.host to your Elasticsearch Host URL.
  5. Set the elasticsearch.api_key to your unrestricted Elasticsearch API key if configured, or the key copied from above.
  6. Set the elasticsearch.verify_certs to false if using a self-managed Elasticsearch deployment with a self signed SSL certificate.

Run the Connector service source code (note you may need to use sudo when running the connector):

make install
make run

You should see a message in the connector service output similar to the following:

[Connector id: <YOUR_CONNECTOR_ID>, index name: <YOUR_INDEX_NAME>] Connector is not configured yet. Finish configuration in Kibana to make it possible to run a sync.

Configure the Connector Settings

Navigate back to the Kibana Deployment page, where you may need to click the Recheck now button for the Connector to be connected.

Once it has verified that the Connector connected, click Next to configure the settings.

On the Configuration page, you should see the Sandfly Server configuration settings.

  1. Enter the Sandfly Server URL, including the /v4 at the end of the URL.
  2. Enter the Sandfly Server Username.
  3. Enter the Sandfly Server Password.
  4. Click to Enable Pass Results if you want both Alerts and Pass results (default is Pass Results disabled). Sandfly recommends enabling Pass Results for a more complete set of data.
  5. Disable Verify SSL Certificate if you are using a self hosted Sandfly Server without a valid SSL certificate, otherwise leave it Enabled.
  6. Enter the number of Days of results history to fetch on a Full Content Sync. The default is 30 days.
  7. Click Save to save the configuration.
  8. Click Next to finish up the Connector configuration.
  9. Click Manage connector to view or make changes to the connector.

If the Associated index was automatically generated or if you created the index yourself, you must configure the index Settings and Mappings before doing any sync operations.

🚧

CAUTION: Do Not Perform a Full Content Sync Operation At This Stage

DO NOT perform a Full Content Sync operation until you configure the index, as this will affect synchronization. Continue with the integration, a full sync will be done later in this process.

Configure the Index Settings and Mappings

In order for the Sandfly data to be correctly ingested and parsed into fields, you will need to configure the Index Settings and Mappings.

Settings and Mappings JSON files are located in the Sandfly Setup distribution in the sandfly-setup/integrations/elasticsearch folder.

Click on the Console tab at the very bottom of the screen to bring up the Dev Tools Console.

Click on Clear this input to clear out any previous console sessions.

Type the following text in the Console: PUT /<YOUR_INDEX_NAME>/_settings and then copy/paste the JSON from the sandfly_elastic_settings.json file below the text so that it looks like this:

PUT /<YOUR_INDEX_NAME>/_settings
{
  "mapping": {
	"total_fields": {
  	"limit": "5000"
	}
  }
}

Click the Run button to send the request, and verify the output is "acknowledged": true.

Type the following text in the Console: PUT /<YOUR_INDEX_NAME>/_mapping and then copy/paste the JSON from the sandfly_elastic_mappings.json file below the text so that it looks something like this (the mappings JSON is very large so not all is included here):

PUT /<YOUR_INDEX_NAME>/_mapping
{
 "properties": {
  "_timestamp": {
   "type": "date"
  },
  "id": {
   "fields": {
	"keyword": {
 	"ignore_above": 256,
 	"type": "keyword"
	}
   },
   "type": "text"
  },

  ...

}

Click the Run button to send the request, and verify the output is "acknowledged": true.

Close the Console and navigate to the Index Management page. Click on the Index name, then select the Settings tab to verify the total_fields.limit is set to 5000, and select the Mappings tab to verify the mappings are configured.

Run a Full Content Sync

Navigate to the Elasticsearch Connectors page and click on your new connector.

Verify the Customized connector indicates Connected in green.

Select the Configuration tab, scroll to the bottom and click the Sync button and select a Full Content sync.

You should see a Sync pending appear in the Overview sync list and should see activity in the Connector service output.

Wait until the Sync operation is complete.

⚠️

WARNING: Full Content Sync Sets the Current State, Deleting Old Data

It is important to note that a Full Content Sync operation will sync the Index to the current state of the Sandfly Server. This means that any documents that are not included in the Sync operation will be deleted automatically. So any results data older than the number of Days of results history to fetch will be deleted and any hosts or SSH Keys not included in the Sync operation will be deleted.

For this reason, Sandfly recommends that you only run a Full Content Sync once after creating the connector, or when you are absolutely sure you want to clean out old data and sync Elasticsearch to the current state of the Sandfly Server.

Schedule an Incremental Content Sync

Navigate to the Elasticsearch Connectors page and click on your new connector.

  1. Select the Scheduling tab.
  2. Click on Incremental content sync to expand the options.
  3. Click the Enabled button to enable the options.
  4. Select your Frequency and then configure the desired times.
  5. Click Save to save and apply the settings.

Dashboards and Reports

An example set of Dashboards is located in the Sandfly Setup distribution in the sandfly-setup/integrations/elasticsearch folder. You can upload the dashboards to your Analytics Dashboards.

Navigate to Stack Management > Kibana > Saved Objects page.

  1. Click on the Import link.
  2. Select the sandfly_elastic_dashboards.ndjson file.
  3. Under Import options
    • Select Check for existing objects, and
    • Select Automatically overwrite conflicts.
  4. Click the Import button and verify the objects to be imported.
  5. Click Done.

There are three data view objects that must be modified to point to your specific index.

Type "View" in the search box and you should see three data view objects:

  • Sandfly-View-Results
  • Sandfly-View-Hosts
  • Sandfly-View-SSH-Keys.

Click on the view to inspect the index pattern.

  1. Click the Edit button.
  2. Change the Index pattern to match your Sandfly Index.
  3. Click Save.
  4. Click Confirm if prompted.

Do the same for all three views.

Navigate to the Dashboards page to review the dashboards.

Updated 3 months ago