Install Ubuntu 18.04 (see detail right at the bottom of this post with the basic Ubuntu install screenshots)
Once Ubuntu is installed ssh to the box and run the following:
Run sudo -i to be in perm sudo mode
apt update
apt upgrade
Create snapshot here (call it your Ubuntu base install)
sudo -i (if you rebooted since doing the updates)
Run the following:
add-apt-repository -y ppa:webupd8team/java
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
apt install apt-transport-https
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
apt update
apt install openjdk-11-jre-headless
apt install elasticsearch; sudo apt install kibana; sudo apt install logstash
Edit the Kibana configuration file:
nano /etc/kibana/kibana.yml
- server.port: 5601
- server.host: "0.0.0.0"
Enable and start the services:
systemctl enable kibana.service
systemctl start kibana
systemctl status kibana
You want to see the highlighted part, then you know that is working:
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; disabled; vendor preset: enabled)
Active: active (running) since Thu 2020-07-02 09:19:37 UTC; 21s ago
Main PID: 5432 (node)
Tasks: 11 (limit: 4915)
CGroup: /system.slice/kibana.service
└─5432 /usr/share/kibana/bin/../node/bin/node /usr/share/kibana/bin/../src/cli
If you are running internally with no exposed services, disable the firewall. (or you need to add exclusions):
systemctl stop ufw.service
systemctl disable ufw.service
Test it at this point and you should get the following:
Leave that as is and go back to the SSH Terminal: (still running as sudo -i)
Edit the elasticsearch config file, make the highlighted changes
nano /etc/elasticsearch/elasticsearch.yml
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: localhost
#
# Set a custom port for HTTP:
#
http.port: 9200
#
Enable and start the services:
systemctl daemon-reload
systemctl enable elasticsearch.service
systemctl start elasticsearch.service
systemctl status elasticsearch.service
You should see it saying Running, similar to the Kibana service ..
Give it 2 minutes and then test it..
You should now see this..
And then end up here..
Leave that as it is and head back to the SSH Terminal (still as sudo -i)
Add/Update Logstash plugins: (if you did a default install these file locations should be correct)
Run these one by one….. things to note… some take a while to install and say success updated …..
Some run for a while and say nothing changed… don’t start hacking your install, its all ok… read on..
/usr/share/logstash/bin/logstash-plugin install logstash-codec-sflow
/usr/share/logstash/bin/logstash-plugin update logstash-codec-netflow
/usr/share/logstash/bin/logstash-plugin update logstash-input-udp
/usr/share/logstash/bin/logstash-plugin update logstash-input-tcp
/usr/share/logstash/bin/logstash-plugin update logstash-filter-dns
/usr/share/logstash/bin/logstash-plugin update logstash-filter-geoip
/usr/share/logstash/bin/logstash-plugin update logstash-filter-translate
If you are concerned about the plugins that said nothing installed run this command to see what plugins are install and what versions there are..
bin/logstash-plugin list --verbose
Create snapshot here (call it your Elastic install)
Now we install ElastiFlow: ( still running as sudo -i)
I modified the installs I found on the Internet to install ver 3.5.0.
wget https://github.com/robcowart/elastiflow/archive/v3.5.0.zip
apt install unzip
unzip v3.5.0.zip
cp -r elastiflow-3.5.0/logstash/elastiflow /etc/logstash
cp -r elastiflow-3.5.0/logstash.service.d/elastiflow.conf /etc/systemd/system/logstash.service.d
Edit the Java Virtual Machine config file, change the memory allocation to almost as much ram as you can give the box (mine has 8 gig)
sudo nano /etc/logstash/jvm.options
-Xms6g
-Xmx6g
Edit the piplelines configuration file and add the two highlighted lines, make sure the spacing is exactly the same…
sudo nano /etc/logstash/pipelines.yml
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
- pipeline.id: elastiflow
path.config: "/etc/logstash/elastiflow/conf.d/*.conf"
Now we need to edit the inputs, still ad sudo -I edit the appropriate config file, if you are not using Netflow IP4, then just do a directory listing in the folder to find the right file.(or look below I pasted the directory contents)
cd /etc/logstash/elastiflow/conf.d/
nano 10_input_netflow_ipv4.logstash.conf
Change the value to true if you want DNS lookups to be done..then add your internal nameserver..
Environment="ELASTIFLOW_RESOLVE_IP2HOST=true"
Environment="ELASTIFLOW_NAMESERVER=10.10.10.11"
Further below set the host to 0.0.0.0 to listen to all inputs, unless you only send from one host and want to tie it to that one specific host..then also change the port to whatever port you are sending Netflows on
# Netflow - IPv4
Environment="ELASTIFLOW_NETFLOW_IPV4_HOST=0.0.0.0"
Environment="ELASTIFLOW_NETFLOW_IPV4_PORT=9995"
# Netflow - IPv6
Environment="ELASTIFLOW_NETFLOW_IPV6_HOST=[::]"
Environment="ELASTIFLOW_NETFLOW_IPV6_PORT=52055"
# Netflow - UDP input options
Environment="ELASTIFLOW_NETFLOW_UDP_WORKERS=4"
Environment="ELASTIFLOW_NETFLOW_UDP_QUEUE_SIZE=4096"
Environment="ELASTIFLOW_NETFLOW_UDP_RCV_BUFF=33554432"
# Netflow timestamp options
Environment="ELASTIFLOW_NETFLOW_LASTSW_TIMESTAMP=false"
Environment="ELASTIFLOW_NETFLOW_TZ=UTC"
# sFlow - IPv4
Environment="ELASTIFLOW_SFLOW_IPV4_HOST=0.0.0.0"
Environment="ELASTIFLOW_SFLOW_IPV4_PORT=6343"
# sFlow - IPv6
Environment="ELASTIFLOW_SFLOW_IPV6_HOST=[::]"
Here is the directory listing..
/etc/logstash/elastiflow/conf.d/
Dir…
10_input_ipfix_ipv4.logstash.conf 10_input_sflow_ipv6.logstash.conf.disabled 20_filter_90_post_process.logstash.conf
10_input_ipfix_ipv6.logstash.conf.disabled 20_filter_10_begin.logstash.conf 30_output_10_single.logstash.conf
10_input_netflow_ipv4.logstash.conf 20_filter_20_netflow.logstash.conf 30_output_20_multi.logstash.conf.disabled
10_input_netflow_ipv6.logstash.conf.disabled 20_filter_30_ipfix.logstash.conf
10_input_sflow_ipv4.logstash.conf 20_filter_40_sflow.logstash.conf
Enable and start the services:
systemctl daemon-reload
systemctl enable logstash.service
systemctl start logstash.service
Now at this point it’s a good idea to go to your device that will send Netflows and enable the sending… once that is done go back to your SSH Terminal and run the following command (still as sudo -i)
tail -f /var/log/logstash/logstash-plain.log
It will take a while and start running up a load of things, if you see anything about error….then something did not work as planned..
Now….admittedly… I have no idea where I finally managed to find this file or whether its really needed in this version of elastic, I guess I could do a full new install and see if I leave it out the patterns are downloaded automatically… I found the file somewhere, used SSH to copy it to the Ubuntu machine. If you want the file message me and I can drop it in email, its 1.3 meg, or try without it and tell me if it works
Copy elastiflow.index_pattern.json file to the home folder of admin_it, then used the command below to create the Kibana Indexes.
curl -X POST http://10.10.10.92:5601/api/saved_objects/index-pattern/elastiflow-* -H "Content-Type: application/json" -H "kbn-xsrf: true" -d @/home/admin_it/elastiflow.index_pattern.json
Almost there, now connect to Kibana Interface
http://(Machine IP Address):5601
Download the 7.5 dashboard from Rob Cowart’s Github page..
Add the Dashboard by importing it, go to Kibana/Stack Management/Saved objects… then on the top right click on import, browse to and select the downloaded file, click on import and it should come back and say success...
By Now in the tail log you should see something like this…
Give it a few minutes and you should see the flows appear in Kibana under the index patterns.
