This blog is part of a series. Refer to the links below for the other posts in this series.

Part 1: Installing Ubuntu
Part 2: Installing Elastic Stack
Part 3: Install ElastiFlow

In parts 1 and 2 of this tutorial, we installed the Ubuntu server and Elastic Stack (ELK Stack). Now we are ready to install and configure ElastiFlow.

Before beginning, I recommend setting up at least one network device to start sending logs to the server. In my environment, I configured my pfSense firewall to send IPv4 flows using port 9995. It is important that you make note of the port you set up in your environment, as we will need to configure ElastiFlow to receive them as part of this tutorial.

The steps below are based on the directions found in ElastiFlow GitHub site. I’ve just expanded upon them and given you the command relevant to the Ubuntu and Elastic Stack 6.3 install we performed in parts 1 and 2. The instructions here are for ElastiFlow 3.x

Set JVM heap size

It is recommended to set the JVM heap size to at least 2GB. If you are going to be doing DNS lookups then 4GB is recommended.

Open the jvm.options for Logstash to set the heap size

sudo nano /etc/logstash/jvm.options

Edit the Xms and Xmx sizes in the jvm.options configuration

-Xms4g 

-Xmx4g

Add and Update Required Logstash plugins

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns

Copy the pipeline files to the Logstash configuration path

Create a temp folder to hold install files

mkdir flowtemp

Navigate to the temp folder you just created

cd flowtemp

Download Elastiflow install files

wget https://github.com/robcowart/elastiflow/archive/master.zip

Install unzip, so you can extract the archive file you just downloaded

sudo apt-get install -y unzip

Unzip the Elastiflow files

unzip master.zip

Copy ElastiFlow configuration files to the Logstash directory

sudo cp -a elastiflow-master/logstash/elastiflow/. /etc/logstash/elastiflow/

Setup environment variable helper files

Copy the elastiflow.conf to systemd

sudo cp -a elastiflow-master/logstash.service.d/. /etc/systemd/system/logstash.service.d/

Add the ElastiFlow pipeline to pipelines.yml

Open the Logstash pipline configuration file for editing.

sudo nano /etc/logstash/pipelines.yml

Add the two line below to the bottom of the pipelines.yml file

- pipeline.id: elastiflow   

path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

Configure inputs

Open the elastiflow.conf file for editing.

sudo nano /etc/systemd/system/logstash.service.d/elastiflow.conf

The items you set here will be unique to your environment and setup. In my environment, I set the following:

ELASTIFLOW_NETFLOW_IPV4_HOST=<The Server’s IP Address>
ELASTIFLOW_NETFLOW_IPV4_PORT=9995

Remember 9995 is the port I configured the network equipment to send flows on.

I also set ELASTIFLOW_RESOLVE_IP2HOST to true and set my DNS server in ELASTIFLOW_NAMESERVER so that the dashboards will attempt to resolve the DNS names instead of just displaying IP Address. There is a performance hit for this, but since it is just my lab network, it should not be a problem.

Ensure that the port for the incoming flows is open, on the firewall, so that Logstash is able to receive them.

sudo ufw allow from <IP Address> to any port 9995 proto tcp

Create logstash system startup script

sudo /usr/share/logstash/bin/system-install

Reload systemd manager configuration and start logstash

sudo systemctl daemon-reload 

sudo systemctl start logstash

Run the command below to check that logs are being received.

tail -f /var/log/logstash/logstash-plain.log

You should see log entries scrolling up the screen. Logstash can take some time to start so wait a few minutes after running the command. If after a little bit, it is just sitting there doing nothing, then either flows are not being sent or something is wrong with your configuration. If something is not configured correctly, you should see the error listed in the log. You can ignore any errors about there being nothing in the “/etc/logstash/conf.d/*.conf” folder. This is because we added the ElastiFlow to a second pipeline, so unless you previously setup anything on this server, than that folder should be empty.

Note: If using Netflow v9 or IPFIX you will likely see warning messages related to the flow templates not yet being received. They will disappear after templates are received from the network devices, which should happen every few minutes. Some devices can take a bit longer to send templates. Fortinet in particular send templates rather infrequently.

Hit Ctrl-C to exit from log tail

Setup Kibana

Assuming you are still in the flowtemp directory, run the command below to import the ElastiFlow indexes.

curl -X POST http://<Your Server's IP Address>:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @elastiflow-master/kibana/elastiflow.index_pattern.json

1. On your local machine download the ElastiFlow dashboards. Right-click the link below choose save as https://github.com/robcowart/elastiflow/raw/master/kibana/elastiflow.dashboards.json
2. Open your web browser and open the Kibana site.
3. Navigate to Management > Advanced Settings
4. Search for and set the recommended settings listed below. For details and additional information on what these are, refer to the ElastiFlow documentation.
   doc_table:highlight -> false
   filters:pinnedByDefault -> true
   state:storeInSession -> Storage
   truetimepicker:quickRanges -> see link
5. Navigate to Saved Objects and Import elastiflow.dashboards.json file you downloaded in step 1.

Once dashboard import completes you are done. You can now navigate to the Dashboard page in Kibana and start exploring the different visualizations. You can also check out the ElastiFlow Dashboard Documentation.