Collecting and Visualizing vSphere Performance Metrics with PowerCLI, InfluxDB and Grafana

    posted by Mike Nisk on Apr 08, 2016

    Introduction

    In this guide, I will get you up and running with InfluxDB and Grafana on CentOS 7. Once you complete this setup, you can start collecting and displaying Realtime performance metrics from your favorite API. This does not currently support VSAN (only NFS and VMFS). If you need VSAN stats check out the amazing appliance from SexiGraf.



    Note: This can also be deployed 100% on Windows. See my How To Guide - Compiling InfluxDB on Windows.


    Motivation

    If you are interested in gathering VMware vSphere performance metrics, then you can use my vFlux-IOPS.ps1 and vFlux-Compute.ps1 scripts to feed InfluxDB (available on github). My PowerCLI scripts, along with InfluxDB and Grafana create a vSphere performance monitoring kit that I refer to as vFlux Stats.


    System Requirements

    1 Virtual Machine (minimum)
    CentOS 7 (requires vSphere 5.5 or later)
    2 vCPU
    8GB RAM
    40GB Harddrive


    Note: In this exercise, we will use a single disk for simplicity. In a high performance deployment, you will want dedicated disks for InfluxDB (i.e. Data and WAL) and most likely you would be running Grafana on an additional CentOS VM.


    CentOS 7 - Initial Setup

    Deploy a CentOS 7 VM (i.e. from template or ISO) then follow the steps below to configure an IP address, hostname, and perform typical OS updates, etc.


    1. Login as root
    2. Configure an IP address using the nmtui text-based user interface tool
    3. Perform a systemctl restart network
    4. Test your success so far ping google.com
    5. Install VMware Tools with yum install open-vm-tools
    6. Update the OS and existing packages with yum update
    7. [optional] To allow root login via SSH, use the vi editor to uncomment the root login restriction. We will do this with vi /etc/ssh/sshd_config
    8. [optional] If you made the change in #7 above, perform a systemctl restart sshd.service
    9. If desired, confirm SSH access (i.e. with putty)



    Note: If you need to resize your disk, consult http://kb.vmware.com/kb/1006371


    Installing InfluxDB on CentOS 7

    To install InfluxDB on CentOS, there are a few options. For this example, we will create a yum repository and install from there.



    Add InfluxDB to yum
    Just paste this entire snippet into your putty session:

    cat <<EOF | sudo tee /etc/yum.repos.d/influxdb.repo
    [influxdb]
    name = InfluxDB Repository - RHEL \$releasever
    baseurl = https://repos.influxdata.com/rhel/\$releasever/\$basearch/stable
    enabled = 1
    gpgcheck = 1
    gpgkey = https://repos.influxdata.com/influxdb.key
    EOF
    



    Install InfluxDB

    sudo yum install influxdb
    



    We now have several binaries added to /usr/bin.

    influx
    influxd
    influx_inspect
    influx_stress
    influx_tsm
    

    influx is the CLI tool to manage things.
    influxd is the actual binary that runs the program itself.



    Allow InfluxDB through Firewall

    firewall-cmd --permanent --zone=public --add-port=8086/tcp
    firewall-cmd --permanent --zone=public --add-port=8083/tcp
    firewall-cmd --reload
    



    Configure InfluxDB
    The default config file will be at /etc/influxdb/influxdb.conf. However, you can create a new config file to modify if desired.

    influx config > influxdb.generated.conf
    

    Note: You can then use the -config parameter to launch InfluxDB. For example, influxd -config influxdb.conf



    Configure InfluxDB for Automatic start-up

    systemctl enable influxdb.service
    



    Start InfluxDB

    sudo service influxdb start
    



    The InfluxDB Web Interface

    Once InfluxDB is up and running, connect to it using a web browser.

    http://<ip address>:8083
    



    Using the influx CLI

    To interact with your installation of InfluxDB (i.e. create users, databases, etc.) perform the following:

    1. SSH to your InfluxDB VM
    2. Change directory to /usr/bin
    3. Type influx and hit enter
    [root@centflux02 bin]# influx
    Visit https://enterprise.influxdata.com to register for updates, InfluxDB server management, and monitoring.
    Connected to http://localhost:8086 version 0.11.1
    InfluxDB shell 0.11.1
    >
    



    Create InfluxDB Users

    For this example, create a user called esx with a password of esx. Type the following into your influx CLI session and press enter.

    CREATE USER esx WITH PASSWORD 'esx' WITH ALL PRIVILEGES
    

    Tip: Influx commands only return interactive messages on failure. So after hitting enter above, if you get no feedback, this is good.



    Note: For full details on InfluxDB authentication:
    https://docs.influxdata.com/influxdb/v0.11/administration/authentication_and_authorization/



    Create InfluxDB Database

    For this example, create a database called iops and another called compute.

    CREATE DATABASE iops
    CREATE DATABASE compute
    



    Note: We will use these databases in the next steps.


    Installing Grafana on CentOS 7

    To install Grafana on CentOS, there are a few options. For this example, we will create a yum repository and install from there. This is similar to how we got InfluxDB up and running.



    Create a new file in /etc/yum.repos.d/grafana.repo and add the following:

    [grafana]
    name=grafana
    baseurl=https://packagecloud.io/grafana/stable/el/6/$basearch
    repo_gpgcheck=1
    enabled=1
    gpgcheck=1
    gpgkey=https://packagecloud.io/gpg.key https://grafanarel.s3.amazonaws.com/RPM-GPG-KEY-grafana
    sslverify=1
    sslcacert=/etc/pki/tls/certs/ca-bundle.crt
    



    Install Grafana

    sudo yum install grafana
    



    Configure Grafana

    If desired, you can modify the Grafana config file at /etc/grafana/grafana.ini (i.e. to change the default port of 3000, etc.).



    Configure Grafana for automatic start-up

    sudo systemctl enable grafana-server.service
    



    Start Grafana

    systemctl daemon-reload
    systemctl start grafana-server
    systemctl status grafana-server
    



    Allow Grafana through firewall

    firewall-cmd --permanent --zone=public --add-port=3000/tcp
    firewall-cmd --reload
    



    Check Grafana Service Status

    systemctl status grafana-server
    



    The Grafana Web Interface
    The default login is admin admin. Here, you can point Grafana to your InfluxDB databases. Later, we will also create graphs of our performance data from this interface.

    http://<ip address>:3000
    



    Checking your work
    Before calling this done, let's confirm that both InfluxDB and Grafana are configured for auto-start.

    systemctl list-unit-files | grep 'influx\|grafana'
    



    The output from above should look similar to:

    grafana-server.service                      enabled
    influxd.service                             enabled
    influxdb.service                            enabled
    



    Next Steps
    The next step is to start loading up the database with some performance stats from your VMware environment. For this task you can use my vFlux Stats kit. I provide the PowerCLI scripts to run against your vCenter. The scripts interrogate VMs and ESXi hosts for realtime performance data. The data is then written to InfluxDB and served up by Grafana.



    Download the vFlux Stats Kit (PowerCLI scripts):
    https://github.com/vmkdaily/vFlux-Stats-Kit



    Supporting Documentation
    https://docs.influxdata.com/influxdb/v0.11/introduction/installation/
    http://docs.grafana.org/installation/rpm/



    Who to Follow
    @InfluxDB
    @grafana
    @vmkdaily


    Example Graphs

    vFlux Stats Example NFS IOPS Dashboard



    vFlux Stats Example Block IOPS Dashboard



    vFlux Stats Example vCPU Dashboard



    Category:InfluxDB Tags: InfluxDB Grafana PowerCLI vSphere ESXi Performance stats



    How To Guide - Compiling InfluxDB on Windows

    posted by Mike Nisk on Apr 03, 2016

    Introduction

    This is a quick guide showing how to compile InfluxDB on Windows.


    Requirements

    First, we need to download and install the Windows versions of go, git and hg.


    Note: InfluxDB requires Go 1.4.3

    Downloads
    https://golang.org/dl/
    https://git-scm.com/downloads
    http://tortoisehg.bitbucket.org/download/


    Compile InfluxDB

    Open an elevated command prompt and enter the following:

    cd c:\go
    mkdir projects
    set "GOPATH=C:\Go\projects"
    cd %gopath%
    go get github.com/influxdata/influxdb
    cd src\github.com\influxdata\influxdb
    go get -u -f ./...
    go build ./...
    


    Create the InfluxDB Config File

    cd c:\go\projects\bin
    influxd config > influxdb.generated.conf
    copy influxdb.generated.conf influxdb.conf
    


    Modify the Config File

    start wordpad influxdb.conf 
    


    Launch InfluxDB

    influxd –config influxdb.conf
    


    Next Steps

    Other things you might try now are:

    1. Configure this as a service using NSSM.
    2. Gather VMware vSphere stats and write them to InfluxDB vFlux-Stats-Kit on github


    Source

    The approach documented above reflects the new path locations (i.e. influxdata instead of influxdb) and is based on:
    http://stackoverflow.com/questions/26116711/how-to-install-inflxdb-in-windows


    Category:InfluxDB Tags: Windows golang NSSM InfluxDB



    PowerCLI One-Liner to Get VM Tags

    posted by Mike Nisk on Mar 29, 2016

    So, here's an easy one. Sometimes you need a list of a virtual machine tags from vSphere.

    Get-VM <Name of VM> | Select Name,@{N="Tags";E={((Get-TagAssignment -Entity $_ | select -ExpandProperty Tag).Name -join ",")}}
    



    If you want to output that to a csv, then you can pipe that to Export-Csv:

    Get-VM <Name of VM> | Select Name,@{N="Tags";E={((Get-TagAssignment -Entity $_ | select -ExpandProperty Tag).Name -join ",")}} | Export-Csv -NoTypeInformation c:\temp\tags.csv
    

    Note: Optionally, you can pipe to Format-List (i.e. fl *) or even easier, pipe to Out-GridView.


    That's all for now. Enjoy!


    Category:PowerCLI Tags: vSphere Tags PowerCLI



    Check For HA Events with PowerCLI

    posted by Mike Nisk on Mar 29, 2016

    Intro

    You can check for vSphere HA events in a number of ways. I find that using PowerCLI is the best.

    How To

    Just hit enter to accept the defaults. Optionally, you can enter the number of hours and events to search on.

    # CheckForHAEvents.ps1
    # Quick one-liner to check for recent HA events.
    # Note:  Adjust "MaxSamples" to a higher number to search further back
    # Based on:
    # http://www.jonathanmedd.net/2012/03/which-vms-restarted-after-a-vsphere-ha-event.html
    
    #More Info:
    #http://www.yellow-bricks.com/2012/09/20/a-host-has-failed-which-vms-were-impacted-and-restarted-by-ha/
    
    Write-Host "`nWelcome to vSphere HA Checker`n" -ForegroundColor Yellow
    Write-Host "This script:"
    Write-Host "- Checks for any recent HA Events"
    Write-Host "- If any VMs were hard rebooted by HA, they will show up here"
    Write-Host "- User provides the time range (how far back in the logs to search)"
    Write-Host "- User provides max number of VI events to review`n"
    
    $range = Read-Host "Enter the time range in hours (default 24 hours)"
    If(!($range)){$range = 24}
    
    $MaxSamples = Read-Host "Enter the max number of events to search (default 100000)"
    If(!($MaxSamples)){$MaxSamples = 100000}
    
    
    Write-Host "`nParsing vSphere logs..." -ForegroundColor Yellow
    $Results = Get-VIEvent -start (Get-Date).addhours(-$range) -MaxSamples $MaxSamples | Where-Object {$_.eventtypeid -eq "com.vmware.vc.ha.VmRestartedByHAEvent"} | Select ObjectName,CreatedTime
    
    ## Echo results to screen
    If(!$Results){
        Write-Host "No HA Events found in the past $range hour(s)`n" -ForegroundColor Green
        }
    Else {
        Write-Host "HA Events Detected on:"
        $Results
        }
    

    Note: If any VMs have been hard rebooted by vSphere HA, they will be listed.


    Category:PowerCLI Tags: vSphere HA Availability PowerCLI