Collecting and Visualizing vSphere Performance Metrics with PowerCLI, InfluxDB and Grafana

posted by Mike Nisk on Apr 08, 2016

Introduction

In this guide, I will get you up and running with InfluxDB and Grafana on CentOS 7. Once you complete this setup, you can start collecting and displaying Realtime performance metrics from your favorite API. This does not currently support VSAN (only NFS and VMFS). If you need VSAN stats check out the amazing appliance from SexiGraf.



Note: This can also be deployed 100% on Windows. See my How To Guide - Compiling InfluxDB on Windows.


Motivation

If you are interested in gathering VMware vSphere performance metrics, then you can use my vFlux-IOPS.ps1 and vFlux-Compute.ps1 scripts to feed InfluxDB (available on github). My PowerCLI scripts, along with InfluxDB and Grafana create a vSphere performance monitoring kit that I refer to as vFlux Stats.


System Requirements

1 Virtual Machine (minimum)
CentOS 7 (requires vSphere 5.5 or later)
2 vCPU
8GB RAM
40GB Harddrive


Note: In this exercise, we will use a single disk for simplicity. In a high performance deployment, you will want dedicated disks for InfluxDB (i.e. Data and WAL) and most likely you would be running Grafana on an additional CentOS VM.


CentOS 7 - Initial Setup

Deploy a CentOS 7 VM (i.e. from template or ISO) then follow the steps below to configure an IP address, hostname, and perform typical OS updates, etc.


  1. Login as root
  2. Configure an IP address using the nmtui text-based user interface tool
  3. Perform a systemctl restart network
  4. Test your success so far ping google.com
  5. Install VMware Tools with yum install open-vm-tools
  6. Update the OS and existing packages with yum update
  7. [optional] To allow root login via SSH, use the vi editor to uncomment the root login restriction. We will do this with vi /etc/ssh/sshd_config
  8. [optional] If you made the change in #7 above, perform a systemctl restart sshd.service
  9. If desired, confirm SSH access (i.e. with putty)



Note: If you need to resize your disk, consult http://kb.vmware.com/kb/1006371


Installing InfluxDB on CentOS 7

To install InfluxDB on CentOS, there are a few options. For this example, we will create a yum repository and install from there.



Add InfluxDB to yum
Just paste this entire snippet into your putty session:

cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
[influxdb]
name = InfluxDB Repository - RHEL \$releasever
baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
enabled = 1
gpgcheck = 1
gpgkey = https://repos.influxdata.com/influxdb.key
EOF



Install InfluxDB

sudo yum install influxdb



We now have several binaries added to /usr/bin.

influx
influxd
influx_inspect
influx_stress
influx_tsm

influx is the CLI tool to manage things.
influxd is the actual binary that runs the program itself.



Allow InfluxDB through Firewall

firewall-cmd --permanent --zone=public --add-port=8086/tcp
firewall-cmd --permanent --zone=public --add-port=8083/tcp
firewall-cmd --reload



Configure InfluxDB
The default config file will be at /etc/influxdb/influxdb.conf. However, you can create a new config file to modify if desired.

influx config > influxdb.generated.conf

Note: You can then use the -config parameter to launch InfluxDB. For example, influxd -config influxdb.conf



Configure InfluxDB for Automatic start-up

systemctl enable influxdb.service



Start InfluxDB

sudo service influxdb start



The InfluxDB Web Interface

Once InfluxDB is up and running, connect to it using a web browser.

http://<ip address>:8083



Using the influx CLI

To interact with your installation of InfluxDB (i.e. create users, databases, etc.) perform the following:

  1. SSH to your InfluxDB VM
  2. Change directory to /usr/bin
  3. Type influx and hit enter
[root@centflux02 bin]# influx
Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
Connected to http://localhost:8086 version 0.11.1
InfluxDB shell 0.11.1
>



Create InfluxDB Users

For this example, create a user called esx with a password of esx. Type the following into your influx CLI session and press enter.

CREATE USER esx WITH PASSWORD 'esx' WITH ALL PRIVILEGES

Tip: Influx commands only return interactive messages on failure. So after hitting enter above, if you get no feedback, this is good.



Note: For full details on InfluxDB authentication:
https://docs.influxdata.com/influxdb/v0.11/administration/authentication_and_authorization/



Create InfluxDB Database

For this example, create a database called iops and another called compute.

CREATE DATABASE iops
CREATE DATABASE compute



Note: We will use these databases in the next steps.


Installing Grafana on CentOS 7

To install Grafana on CentOS, there are a few options. For this example, we will create a yum repository and install from there. This is similar to how we got InfluxDB up and running.



Create a new file in /etc/yum.repos.d/grafana.repo and add the following:

[grafana]
name=grafana
baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt



Install Grafana

sudo yum install grafana



Configure Grafana

If desired, you can modify the Grafana config file at /etc/grafana/grafana.ini (i.e. to change the default port of 3000, etc.).



Configure Grafana for automatic start-up

sudo systemctl enable grafana-server.service



Start Grafana

systemctl daemon-reload
systemctl start grafana-server
systemctl status grafana-server



Allow Grafana through firewall

firewall-cmd --permanent --zone=public --add-port=3000/tcp
firewall-cmd --reload



Check Grafana Service Status

systemctl status grafana-server



The Grafana Web Interface
The default login is admin admin. Here, you can point Grafana to your InfluxDB databases. Later, we will also create graphs of our performance data from this interface.

http://<ip address>:3000



Checking your work
Before calling this done, let's confirm that both InfluxDB and Grafana are configured for auto-start.

systemctl list-unit-files | grep 'influx\|grafana'



The output from above should look similar to:

grafana-server.service                      enabled
influxd.service                             enabled
influxdb.service                            enabled



Next Steps
The next step is to start loading up the database with some performance stats from your VMware environment. For this task you can use my vFlux Stats kit. I provide the PowerCLI scripts to run against your vCenter. The scripts interrogate VMs and ESXi hosts for realtime performance data. The data is then written to InfluxDB and served up by Grafana.



Download the vFlux Stats Kit (PowerCLI scripts):
https://github.com/vmkdaily/vFlux-Stats-Kit



Supporting Documentation
https://docs.influxdata.com/influxdb/v0.11/introduction/installation/
http://docs.grafana.org/installation/rpm/



Who to Follow
@InfluxDB
@grafana
@vmkdaily


Example Graphs

vFlux Stats Example NFS IOPS Dashboard



vFlux Stats Example Block IOPS Dashboard



vFlux Stats Example vCPU Dashboard



Category:InfluxDB Tags: InfluxDB Grafana PowerCLI vSphere ESXi Performance stats