Using Remote Elasticsearch for Sensei Reporting
Sensei provides IT administrators with the option of storing reporting data using either Elasticsearch or a MongoDB database depending on the organization’s firewall hardware resources. Elasticsearch is the leading scalable open-source enterprise search engine designed to operate in real-time in distributed environments. MongoDB is a scalable document database with flexible querying and indexing.
If the firewall has enough memory, 8GB or more RAM, and a modern i3 CPU or later, Sensei will select and install an Elasticsearch instance for its database back end. When the amount of memory is 2 to 4GB and the CPU is somewhat weaker, Sensei will automatically install a local MongoDB database on the target system. Both databases are all installed locally during the Sensei’s initial configuration wizard.
Starting with the Sensei 1.5 release, IT administrators can also completely offload the reporting database to a remote Elasticsearch instance, either in the cloud or as an on-premise custom Elasticsearch system. The following post will show how to configure Sensei with a remote Elasticsearch instance.
Remote Elasticsearch can be deployed in two ways:
- Completely offload reporting data to a remote database. This option does not require a local database to be installed with the firewall.
- Use a local database and simultaneously stream a copy of the reporting data to a remote server.
The first option lets users install Sensei even on inexpensive hardware devices with RAM at 1GB or less. Follow the steps below to configure options one and two.
Completely Offload reporting to a Remote Elasticsearch Instance
NOTE: If you have SOHO or higher Sensei paid subscription, we recommend that you install your key before proceeding with the initial configuration wizard since License will activate a feature that will enable you to have central reporting for many firewalls from a single Elasticsearch instance. Otherwise, a single remote ES instance can be used with a single firewall.
- During the Configuration Wizard, after Hardware Check step, you’ll be provided with the option to use a remote Elasticsearch database:
- Enter the Database URI information: (URI example – http://elasticsearch_server_ip:9200 or https://elasticsearch_server_ip:9200).
- Enter the username and password.
- To configure Elasticsearch with a username and password see:
- To check connections and create indexes in the remote Elasticsearch instances click on “Install Database & Proceed.”
- The Wizard will advance if everything is correct.
- Sensei will store the Report Data in the remote Elasticsearch instance with the configuration.
- No Report Data will be stored locally, all data will be stored in the remote Elasticsearch database.
- Note: Database URI still could be used even if Elasticsearch was configured without the username and password.
Stream Reporting Data to a Remote Elasticsearch Instance
The following option requires SOHO or higher Sensei paid subscriptions.
- Go to: Configuration > Reporting & Data > Stream Reporting Data to External Elasticsearch”
- Activate Enabled.
- The Database URI information: (URI example – http://elasticsearch_server_ip:9200 or https://elasticsearch_server_ip:9200).
- To check the connection and create indexes in the remote Elasticsearch instance click on “Check External Database & Create Indexes”
- The Wizard will advance if everything is correct.
- Sensei will store the reporting data for both Local and Remote Databases with this configuration.
- Note: Database URI still could be filled up even if Elasticsearch was configured without the username and password.
Configuring Kibana to Visualize Sensei Reporting Data
General Overview and Architecture:
Basically, you’ll be creating visualizations from indexes created by Sensei. It might be helpful if we take a bit deep dive into how Sensei is organizing its indexes on the remote Elasticsearch server
Sensei creates 6 different indexes which rotate every day:
- [HOSTUUID]_conn_[date] -> For all TCP and UDP connections
- [HOSTUUID]_http_[date] -> For all Http connections
- [HOSTUUID]_tls_[date] -> For all TLS/HTTP connections
- [HOSTUUID]_alert_[date] -> For all blocked connections
- [HOSTUUID]_dns_[date] -> For all DNS connections
- [HOSTUUID]_sip_[date] -> For all SIP connections
Date and HOSTUUID are variable. Date has the YYYYMMDD format; and HOSTUUID is the unique identifier for the machine on which the Sensei engine is running and will only be used if your subscription is SOHO or higher tiers.
Deployments on SOHO and above subscription tiers will have the HOSTUUID information. So it is highly recommended to activate your license before configuring your remote ES instance.
Free and Home tiers will have indexes with [indextype]_[date] format.
To reach the Host UUID information, please head to Sensei GUI > Configuration > About > Host Unique Identifier.
To use Sensei Reporting data in Kibana, Sensei’s prefix must be added to the Kibana index pattern: Open Kibana > Settings > Index Pattern
- Click on Create Index Pattern and
- Paste Host Unique Identifier (To reach The prefix > Sensei GUI > Configuration > About > Host Unique Identifier).
- When you paste Host Unique Identifier, you will see an index list. Sensei creates 6 different indexes. They are:
- “[HOSTUUID]_conn_[date]”; For all TCP and UDP connections
- “[HOSTUUID]_sip_[date]”; For all SIP connections
- “[HOSTUUID]_dns_[date]”; For all DNS connections
- “[HOSTUUID]_http_[date]”; For all Http connections
- “[HOSTUUID]_tls_[date]”; For all https connections
- “[HOSTUUID]_alert_[date]”; For all blocked connections
If you want your visualizations to work for all times:
Add _[conn,sip,dns,http,tls,alert]* to your HOSTUUID to have your visualizations work for all times and independent of the index date. Asterisk will match all dates.
Or, if you want your visualizations for a specific day:
Specify the exact date for the index pattern so that the visualizations work for the specified date.
Click Next to continue for the visualization:
- Select Start_Time > then click on Create Index Pattern.
- To create Report Graphics, Open Kibana > Visualizations
- Click on Create New Visualization and select any chart.
- Select Index File.
- To complete configuration, Select Terms from Aggregation select box and appropriate Field name from the field Select box in the right Bucket section.
- To update the chart, click on the Update button.
- Reports can be saved and added to the Dashboard.
Keep in touch for more...
Follow us on Twitter, LinkedIn, Facebook, Youtube or subscribe to our newsletter to claim the first dibs on bleeding edge development: