This blog is part of a series. Refer to the links below for the other posts in this series.

Part 1: Installing Ubuntu
Part 2: Installing Elastic Stack
Part 3: Install ElastiFlow

In parts 1 and 2 of this tutorial, we installed the Ubuntu server and Elastic Stack (ELK Stack). Now we are ready to install and configure ElastiFlow.

Before beginning, I recommend setting up at least one network device to start sending logs to the server. In my environment, I configured my pfSense firewall to send IPv4 flows using port 9995. It is important that you make note of the port you set up in your environment, as we will need to configure ElastiFlow to receive them as part of this tutorial.

The steps below are based on the directions found in ElastiFlow GitHub site. I’ve just expanded upon them and given you the command relevant to the Ubuntu and Elastic Stack 6.3 install we performed in parts 1 and 2. The instructions here are for ElastiFlow 3.x

Set JVM heap size

It is recommended to set the JVM heap size to at least 2GB. If you are going to be doing DNS lookups then 4GB is recommended.

Open the jvm.options for Logstash to set the heap size

sudo nano /etc/logstash/jvm.options

Edit the Xms and Xmx sizes in the jvm.options configuration

-Xms4g 

-Xmx4g

Add and Update Required Logstash plugins

sudo /usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow sudo /usr/share/logstash/bin/logstash-plugin update logstash-input-udp sudo /usr/share/logstash/bin/logstash-plugin update logstash-filter-dns

Copy the pipeline files to the Logstash configuration path

Create a temp folder to hold install files

mkdir flowtemp

Navigate to the temp folder you just created

cd flowtemp

Download Elastiflow install files

wget https://github.com/robcowart/elastiflow/archive/master.zip

Install unzip, so you can extract the archive file you just downloaded

sudo apt-get install -y unzip

Unzip the Elastiflow files

unzip master.zip

Copy ElastiFlow configuration files to the Logstash directory

sudo cp -a elastiflow-master/logstash/elastiflow/. /etc/logstash/elastiflow/

Setup environment variable helper files

Copy the elastiflow.conf to systemd

sudo cp -a elastiflow-master/logstash.service.d/. /etc/systemd/system/logstash.service.d/

Add the ElastiFlow pipeline to pipelines.yml

Open the Logstash pipline configuration file for editing.

sudo nano /etc/logstash/pipelines.yml

Add the two line below to the bottom of the pipelines.yml file

- pipeline.id: elastiflow   

path.config: "/etc/logstash/elastiflow/conf.d/*.conf"

Configure inputs

Open the elastiflow.conf file for editing.

sudo nano /etc/systemd/system/logstash.service.d/elastiflow.conf

The items you set here will be unique to your environment and setup. In my environment, I set the following:

ELASTIFLOW_NETFLOW_IPV4_HOST=<The Server’s IP Address>
ELASTIFLOW_NETFLOW_IPV4_PORT=9995

Remember 9995 is the port I configured the network equipment to send flows on.

I also set ELASTIFLOW_RESOLVE_IP2HOST to true and set my DNS server in ELASTIFLOW_NAMESERVER so that the dashboards will attempt to resolve the DNS names instead of just displaying IP Address. There is a performance hit for this, but since it is just my lab network, it should not be a problem.

Ensure that the port for the incoming flows is open, on the firewall, so that Logstash is able to receive them.

sudo ufw allow from <IP Address> to any port 9995 proto tcp

Create logstash system startup script

sudo /usr/share/logstash/bin/system-install

Reload systemd manager configuration and start logstash

sudo systemctl daemon-reload 

sudo systemctl start logstash

Run the command below to check that logs are being received.

tail -f /var/log/logstash/logstash-plain.log

You should see log entries scrolling up the screen. Logstash can take some time to start so wait a few minutes after running the command. If after a little bit, it is just sitting there doing nothing, then either flows are not being sent or something is wrong with your configuration. If something is not configured correctly, you should see the error listed in the log. You can ignore any errors about there being nothing in the “/etc/logstash/conf.d/*.conf” folder. This is because we added the ElastiFlow to a second pipeline, so unless you previously setup anything on this server, than that folder should be empty.

Note: If using Netflow v9 or IPFIX you will likely see warning messages related to the flow templates not yet being received. They will disappear after templates are received from the network devices, which should happen every few minutes. Some devices can take a bit longer to send templates. Fortinet in particular send templates rather infrequently.

Hit Ctrl-C to exit from log tail

Setup Kibana

Assuming you are still in the flowtemp directory, run the command below to import the ElastiFlow indexes.

curl -X POST http://<Your Server's IP Address>:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @elastiflow-master/kibana/elastiflow.index_pattern.json

1. On your local machine download the ElastiFlow dashboards. Right-click the link below choose save as https://github.com/robcowart/elastiflow/raw/master/kibana/elastiflow.dashboards.json
2. Open your web browser and open the Kibana site.
3. Navigate to Management > Advanced Settings
4. Search for and set the recommended settings listed below. For details and additional information on what these are, refer to the ElastiFlow documentation.
   doc_table:highlight -> false
   filters:pinnedByDefault -> true
   state:storeInSession -> Storage
   truetimepicker:quickRanges -> see link
5. Navigate to Saved Objects and Import elastiflow.dashboards.json file you downloaded in step 1.

Once dashboard import completes you are done. You can now navigate to the Dashboard page in Kibana and start exploring the different visualizations. You can also check out the ElastiFlow Dashboard Documentation.

This blog is part of a series. Refer to the links below for the other posts in this series.

Part 1: Installing Ubuntu
Part 2: Installing Elastic Stack
Part 3: Install ElastiFlow

In this section, we will cover installing and configuring Elastic Stack 6.x, which will be used to power the ElastiFlow solution. Elastic Stack, often referred to as ELK Stack, consists of Elasticsearch, Logstash, and Kibana. Elasticsearch is a full-text based search engine. Logstash is a data-collection and log-parsing engine, and Kibana is an analytics and visualization platform used to display the ElastiFlow dashboards.

Please note this tutorial is designed for personal or lab environment setups, so we are not going to cover security considerations with the Kibana website. I have provided links below to additional resources if you need to set up restricted access to the Kibana dashboards.

Installing Elastic Stack 6.x

Install Java

Logstash requires Java 8. Java 9 is not supported. So, we need to ensure that we install the proper version.

Add the Oracle Java PPA to apt

sudo add-apt-repository -y ppa:webupd8team/java

Update apt

sudo apt-get update

Install the latest stable version of Oracle Java 8

sudo apt-get install -y oracle-java8-installer

Install Elasticsearch

Import Elasticsearch Signing Key PGP key

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

Confirm apt-transport-https is installed

sudo apt-get install -y apt-transport-https

Add the repository definition to ensure you are getting the latest version

echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list

Update aptsudo apt-get update

Install Elasticsearch

sudo apt-get -y install elasticsearch

Configure Elasticsearch to start automatically when the system boots

sudo /bin/systemctl daemon-reload 

sudo /bin/systemctl enable elasticsearch.service

Start the Elasticsearch service

sudo systemctl start elasticsearch.service

Install Kibana

Update apt

sudo apt-get update

Install Kibana

sudo apt-get -y install kibana

Configure Kibana to start automatically when the system boots

sudo /bin/systemctl daemon-reload 

sudo /bin/systemctl enable kibana.service

Start the Kibana service

sudo systemctl start kibana.service

Install Logstash

Update apt

sudo apt-get update

Install Logstash

sudo apt-get -y install logstash

Configuring Elastic Stack

Before you can install ElastiFlow there are a few things that need to be set up in the Elastic Stack.

Configure Elasticsearch

Open the Elasticsearch configuration file for editing.

sudo nano /etc/elasticsearch/elasticsearch.yml

Edit the network.host entry in the Elasticsearch configuration to block access to Elasticseach from outside the server.

Set – network.host: localhost

Restart the Elasticsearch service to force the changes to take effect.

sudo systemctl restart kibana

Configure Kibana

Open the Kibana configuration file for editing.

sudo nano /etc/kibana/kibana.yml

Edit the server.host entry in the configuration to allow external access to Kibana.
note: As I mentioned that the beginning of this tutorial this will allow anonymous access to the Kibana dashboard. If you need to restrict access, I recommend installing and configuring Nginx.

Set – server.host: <Your Server’s IP Address>

Restart the Kibana service to force the changes to take effect.

sudo systemctl restart kibana

To ensure that you can access the Kibana site externally you will need to open the inbound port on the service’s firewall.

sudo ufw allow from <Your Server's IP Address> to any port 5601 proto tcp

On your local computer open a web browser, navigate to the Kibana URL, and confirm Kibana loads

http://<Your Server’s IP Address>:5601/

If you see a page load similar to the one above, then everything is set up and ready for you to install ElastiFlow.

Part 3: Install ElastiFlow

Would you like to get more information? Get in touch with us here!

ElastiFlow is a great open source NetFlow analyzer that works with Elastic Stack (formerly ELK Stack). Of all the Netflow tools I’ve tested, it has, by far, the best visualizations. However, if like me you aren’t familiar with Elastic Stack the setup can be rather intimidating. In this tutorial, I hope to make it easier for you and everyone who wants to use this awesome tool.

This tutorial is broken up into 4 parts. One to demonstrate how to install Ubuntu server. One for installing and configuring Elastic Stack. One on how to implement ElastiFlow on top of it all. And finally one on how to properly maintain the solution.

Part 1: How to install Ubuntu
Part 2: Installing Elastic Stack
Part 3: Install ElastiFlow

Install and Set up of Ubuntu Server 18.04

I performed my installation of Ubuntu Server using the latest version of 18.04 on a Hyper-V virtual machine (VM), but the instructions will be the same regardless of what hypervisor you are using. The VM had 40GB hard drive and 4GB of RAM.

How to Install Ubuntu 18.04

1. Download Ubuntu server
   Note: I found downloading the BitTorrent was actually much faster than downloading directly from the Ubuntu servers.
2. Create a new VM with a 40GB hard disk and at least 4GB of RAM.
3. Insert the install media and start the VM.
4. Select your preferred language
   
5. Select your keyboard layout
   
6. Choose Install Ubuntu
   
7. At this set, you have the choice to stick with DHCP or use a static address. If you choose to use a static address it is best to set it up now, as it provides a nice easy interface to set it here.
   
8. Configure a proxy address if required
   
9. On the Filesystem setup screen, select Use An Entire Disk
   
10. Press Enter to accept the default disk
    
11. Select Done
    
12. Select Continue
    
13. Create a name for your server and set up the username and password for the root user
    
14. Wait for the installation to complete
    
15. When prompted select Reboot Now
    
16. If prompted eject the installation media from the VM and press Enter to continue booting
    

Setup Ubuntu for ElastiFlow

If you set your IP address during the installation process the only remaining setup action is to install and configure SSH. This will allow you to use a tool like Putty to connect to the server and more easily configure the items in part 2 and 3. (copy and paste FTW!)

1. Log into the VM using the username and password you created during the setup process
2. Install SSH using the command below: sudo add-apt-repository -y ssh
3. Start the SSH service so you can connect to the server: service ssh status
4. On another computer open your preferred SSH client. I recommend PuTTY if you don’t have one.
5. Enter the IP address of your server, set the port to 22, select SSH connection type, and click OK.
6. If you receive a Security Warning, click Yes.
   

You are now all set to start the installation process.

Click here to read Part 2: Installing Elastic Stack

Would you like to get more information? Get in touch with us today!