TIG – Telegraf InfluxDB Grafana

InfluxDB is a time series database designed to handle high write and query loads.

It is an integral component of the TICK or TIG stacks.
TIG: Telegraf + InfluxDB + Grafana
TICK: Telegraf + InfluxDB + Chronograf + Kapacitor

Basically its one of the best stacks on the marktet for capacity and vitality monitoring with huge potential to be extended to your needs.

We are talking here around the TIG-stack.

InfluxDB is within Debian default repositories so the installation is pretty easy, for any other distributions use the official download page.

$ apt install influxdb influxdb-client

Start and enable influxDB service:

$ systemctl start influxdb
$ systemctl enable influxdb

By default InfluxDB.service is listening on TCP port 8086.
Be careful, there is no restriction to localhost, so the port could be world wide accessible if you don’t have a firewall in front of your system.

Start influx-Shell:

 $ influx

Create a telegraf user which will send the metrics data to the InfluxDB:

create database telegraf
create user telegraf with password 'mySecurePW123!'

Set read and write privileges on telegraf database to the telegraf user:

GRANT ALL ON telegraf TO telegraf

Show data for special host ‘system01.us01.example.com’:

SHOW SERIES WHERE "host" = 'system01.us01.example.com'

Attention! Destructive Change: drop all data for host ‘system01.us01.example.com’:

DROP SERIES WHERE "host" = 'system01.us01.example.com'

Telegraf Installation

Download Telegraf

Debian

wget https://dl.influxdata.com/telegraf/releases/telegraf_1.13.0-1_amd64.deb
dpkg -i telegraf_1.13.0-1_amd64.deb

Debian-ARM

Some earlier versions of telegraf were packaged also for ARM (Raspberry Pi or similar systems), for now its unfortunately only a archive, which has to be extracted. Not a big deal, but a kind of annoying manual installation way.

$ wget https://dl.influxdata.com/telegraf/releases/telegraf-1.13.0_linux_armhf.tar.gz
$ tar xf telegraf-1.13.0_linux_armhf.tar.gz
$ useradd telegraf

The configuration of the telegraf service and everything around it the same for both installation methods:

$ systemctl daemon-reload
$ systemctl enable telegraf
$ systemctl start telegraf

Configuration file for telegraf /etc/telegraf/telegraf.conf

Grafana installation

Installation of Grafana is pretty easy – just download the package [DEB or RPM] for your distribution and install it. ARM packages are prebuild.

$ wget https://dl.grafana.com/oss/release/grafana_7.1.5_amd64.deb
$ dpkg -i grafana_7.1.5_amd64.deb

Personally I prefer to run Grafana behind nginx to log and control access. At this step make sure that after the installation of grafana, TCP port 3000 is secured.

Short nginx configuration example:

server {
  listen 80;
  root /usr/share/nginx/www;
  index index.html;

  location /grafana/ {
   proxy_pass http://localhost:3000/;
  }
}

Retention policies for InfluxDB

Create a retention policy for 180 days for the database telegraf.

$ influx
use telegraf

CREATE RETENTION POLICY "180d_policy" ON "telegraf" DURATION 180d REPLICATION 1 DEFAULT

Show all available retention policies for the database telegraf:

SHOW RETENTION POLICIES

Create InfluxDB Backup

In this case, the InfluxDB-database telegraf has to be backed up, to be on the safe side.
InfluxDB also has support for incremental backups.
Snapshotting from the server now creates a full backup if one does not exist and creates numbered incremental backups after that.

$ influxd backup -portable -db telegraf /backup/influxdb-telegraf

Running the backup command with a new directory name, will create a new full backup with that filename, not an incremental backup!

Restore InfluxDB Backup

Be sure to first shut down any running influxd process, before you go on with the restore process:

$ influxd restore -config /etc/influxdb/influxdb.conf /backup/influxdb-telegraf