Data Miner Quick Start Guide

About

SevOne's Data Miner enables you to export data from SevOne NMS for use with other systems. Using the Data Miner command line tool, you can dump data from short-term database tables into a structured format. You can then import the resulting files into the system of your choice.

This document describes the steps to configure the Data Miner parameters and export relevant data. After data export information is configured, a cron job to regularly export data and remove old data from the export directory can be created.

Prerequisites

Before getting started, make sure to have the following ready:

• IP address of the Cluster Leader - IP address or hostname of the Cluster Leader.

• Root access to the Cluster Leader - root user name and password for your SSH session.

• SSH client - SSH client such as PuTTY.

• Database credentials - Database user name and password.

Configuration Steps

Perform the following steps to export short-term table data from SevOne NMS. You can create a cron job to schedule the data export frequency and remove old data according to your needs.

1. Open your SSH client and log into the cluster leader as the root user.

2. If you have not already, create an output directory to export SevOne NMS data to. Enter the following command:

   $ mkdir <full path to output directory>
   Example

   $ mkdir /tmp/dataminer

3. If you have an existing output directory and you have previously exported data to that output directory, make sure to remove any old output files that you no longer need. Enter the following command to remove old output files if necessary:

   If you create a cron job to run Data Miner, make sure to schedule regular removal of old data.

   $ rm /<full path to output directory>/*
   Example

   $ rm /tmp/dataminer/*

4. The example-dataminer.conf file is provided as a template. Before continuing, make a copy of the file, giving it any name that you like.

   $ cp /SevOne/appliance/settings/example-dataminer.conf \

   /SevOne/appliance/settings/<filename>.conf
   Example: /SevOne/appliance/settings/rfs-dataminer.conf

   $ cp /SevOne/appliance/settings/example-dataminer.conf \

   /SevOne/appliance/settings/rfs-dataminer.conf

5. Now open the .conf file for editing. Enter the following command:

   $ nano /SevOne/appliance/settings/<filename>.conf
   Example

   $ nano /SevOne/appliance/settings/rfs-dataminer.conf

    

   <configuration>

   <sevone>

   <plugins>

   <plugin name="jmx"/>

   <plugin name="snmp"/>

   <plugin name="wmi"/>

   </plugins>

   <master>127.0.0.1</master>

   <username>root</username>

   <password></password>

   <database>net</database>

   <port>3306</port>

   <num_threads>4</num_threads>

   <num_selectors>4</num_selectors>

   </sevone>

   <miner>

   <!-- number of threads spawned to select the indicator data -->

   <selectors>8</selectors> <!-- 8 is the optimial for could be tweaked a bit PAS200k-->

   <output_dir>/tmp/dataminer</output_dir>

   <output_prefix>dataminer_</output_prefix>

   <cluster hostname="localhost"/>

   <!-- Alternate syntax

   <cluster>

   <peer ip="peer_ip_here"/>

   </cluster>

   -->

   <device_group name="All Device Groups"/>

   <!--object_group name="Some object group"/-->

   <!--device ip="device_ip_here"/-->

   <!--device name="device_name_here"/-->

   <!--object type="Interface"/-->

   <!--object name="eth0"/-->

   <indicator name="dskAvail"/>

   </miner>

   </configuration>

   You have an option to only include the parameters that you are interested in obtaining the results for. For example, if you specify only an OBJECT, in the results, it will obtain that object in all the devices. If you specify a device, an object, and an indicator, it will provide you with the results for the parameters included i.e. device, object, and indicator.

   To obtain results for all indicators dskAvail from All Device Groups, please see the snippet of the code below. You will notice that the parameters of interest only are included in this example.

   Example

   ...

   ...

   <output_dir>/tmp/dataminer</output_dir>

   <output_prefix>dataminer_</output_prefix>

   <cluster hostname="localhost"/>

   <device_group name="All Device Groups"/>

   <indicator name="dskAvail"/>

   ...

   ...

6. The following are the available parameters and a description of the expected input for each.

   • plugins - Plugins to mine data from. For each plugin that you want to specify, enter <plugin name="plugin_name_here"/> on a separate line. Replace plugin_name_here with one of the following options. You can specify multiple plugins (one plugin per line).
     

     • snmp

     • icmp

     • process

     • http

     • proxyping

     • nbar

     • portshaker

     • dns

     • ipsla

     • deferred

     • wmi

     • jmx

     • vmware

     • webstatus

     • bulkdata

     • mysqldb

     • coc

     • oracledb

     • dynamic

   • master - The IP address or hostname of the cluster leader.

   • username - The user name for the database as defined on your cluster.

   • password - The password for the database as defined on your cluster.

   • database - The name of the database. This should be net.

   • port - The port to connect to the database. This should be 3306.

   • num_threads - The number of CPU threads to mine data. This setting is used only from where the device dataminer is started from. There is no recommended value since dataminer can be run from a NMS peer or a laptop. It depends on how large the data is that one needs to retrieve and the cluster size. You may start with a low number of threads and increase the number until you get good mining performance and system load.

     Examples

     Example 1 - If we have 10 peers and 4 threads, the data will be collected by 4 threads in parallel.

     Example 2 - If we have 8 CPU threads and 10 appliances, then Data Miner will spawn 8 threads, each of them will connect to a peer (8 in parallel, the last 2 will be mined after one of the first 8 has finished its execution ) and spawn num_selectors on that peer to get the data.

   • num_selectors - The number of threads to select indicator data. When each num_threads thread collects data from the short-term table, it can spawn the number of threads specified here for num_selectors to collect data in parallel. This speeds up the overall process. num_selectors is used by each of those threads to get the data from a single peer in parallel. For example, if a peer has 20k rows and 4 selectors are available, each selector will get 5k rows instead of just one working to obtain 20k rows. In a vacuum, this will speedup the process 4 times.

     The product of (num_threads * num_selectors) can be greater than the number of CPU threads. However, we recommend keeping it lower than the number of CPU threads.

   • output_dir - The directory to export SevOne NMS data to.

   • output_prefix - The file name prefix for the data output files.

   • cluster hostname="localhost" - The cluster host name.

   • peer ip="peer_ip_here" - A setting to specify one or more specific peers. Replace peer_ip_here with the IP address or hostname of the peer. Enter a new line, using the format <peer ip="peer_ip_here"/>, for each peer.

   • device_group name="device_group_here" - A setting to specify device groups/device types. Replace device_group_here with the name of the device group/device type. Enter a new line, using the format <device_group name="device_group_here"/>, for each device group/device type.

   • object_group name="object_group_here" - A setting to specify object groups. Replace object_group_here with the name of the object group. Enter a new line, using the format <object_group name="object_group_here"/>, for each object group.

   • device ip="device_ip_here" - A setting to specify the IP addresses of any specific devices that you want to export data for. Replace device_ip_here with the IP address of the device. Enter a new line, using the format <device ip="device_ip_here"/>, for each device.

   • device name="device_name_here" - A setting to specify the names of the devices that you want to export data for. Replace device_name_here with the name of the device. Enter a new line, using the format <device name="device_name_here"/>, for each device.

   • object type="object_type_here" - A setting to specify object types. Replace object_type_here with the name of the object type. Enter a new line, using the format <object type="object_type_here"/>, for each object type.

   • object name="object_name_here" - A setting to specify objects. Replace object_name_here with the name of the object. Enter a new line, using the format <object name="object_name_here"/>, for each object.

   • indicator name="indicator_name_here" - A setting to specify indicators. Replace indicator_name_here with the name of the indicator. Enter a new line, using the format <indicator name="indicator_name_here"/>, for each indicator.

     Syntax on how to add a comment in the configuration file

     Syntax to comment a parameter from the configuration file.

     <!-- Line to comment --> where,
     <!-- = Start of comment
     --> = End of comment

     Example: How to comment parameters in the configuration file

     Lines commented will not be executed.

     Here are a couple of examples on how a parameter can be commented in the configuration file. These parameters will not be executed as they the line is commented out.
     <!--device_group name="device_group_here"/-->
     <!--indicator name="indicator_name_here"/-->

     Example: Parameters in the configuration file to execute

     Parameters should not be commented in the configuration file if you would like to execute them. Here are a couple of examples of parameters that will execute.
     <device_group name="device_group_here"/>
     <indicator name="indicator_name_here"/>

7. Save your changes and close the .conf file.

8. After editing the .conf file, run the dataminer script to export SevOne NMS data to the output directory that you specified. Enter the following command:

   $ dataminer /SevOne/appliance/settings/<filename>.conf
   Example

   $ dataminer /SevOne/appliance/settings/rfs-dataminer.conf

   Job done !

9. Enter the following command to see a list of output files:

   $ ls -la /<full path to output directory>/
   Example

   $ ls -la /tmp/dataminer/

   total 9472

   drwxr-xr-x 2 root root 4096 Dec 11 16:20 .

   drwxr-xr-x 17 root root 4096 Dec 11 16:20 ..

   -rwx------ 1 root root 2008 Dec 11 16:20 dataminer_10.129.12.7_dns_20190108_1808.out

   -rwx------ 1 root root 6337 Dec 11 16:20 dataminer_10.129.12.74_http_20190108_1808.out

   -rwx------ 1 root root 4319972 Dec 11 16:20 dataminer_10.129.12.74_snmp_20190108_1808.out

   -rwx------ 1 root root 2008 Dec 11 16:20 dataminer_10.129.15.16_dns_20190108_1808.out

   -rwx------ 1 root root 6340 Dec 11 16:20 dataminer_10.129.15.16_http_20190108_1808.out

   -rwx------ 1 root root 5337116 Dec 11 16:20 dataminer_10.129.15.16_snmp_20190108_1808.out

10. Notice the format of the file names. For example, the snmp output file name dataminer_10.129.12.7_snmp_20190108_1808.out is made up of the following components:

    • dataminer_ - The output prefix (output_prefix).

    • 10.129.12.7 - The IP address of the SevOne device that the data comes from.

    • snmp - The plugin (plugin name).

    • 20190108 - The date that the output file was created.

    • 1808 - The time that the output file was created.

11. Run the command below to view the contents of the output file.

    $ less /<full path to output files>/<filename>.out
    Example

    $ less /tmp/dataminer/dataminer_10.129.12.7_snmp_20190108_1808.out

     

    deviceId=117|deviceName=sevtest.sevone.com|deviceIp=10.129.15.215|objectName=/-1|indicator=dskAvail|type=GAUGE|pollTime=2019-01-08 14:20:11|pollValue=21532608.000000|pollTimestamp=1546957211.825632

    deviceId=117|deviceName=sevtest.sevone.com|deviceIp=10.129.15.215|objectName=/-1|indicator=dskAvail|type=GAUGE|pollTime=2019-01-08 14:25:11|pollValue=21532600.000000|pollTimestamp=1546957511.763473

    deviceId=117|deviceName=sevtest.sevone.com|deviceIp=10.129.15.215|objectName=/-3|indicator=dskAvail|type=GAUGE|pollTime=2019-01-08 14:20:11|pollValue=21532608.000000|pollTimestamp=1546957211.829637

    deviceId=117|deviceName=sevtest.sevone.com|deviceIp=10.129.15.215|objectName=/-3|indicator=dskAvail|type=GAUGE|pollTime=2019-01-08 14:25:11|pollValue=21532600.000000|pollTimestamp=1546957511.764789

    deviceId=117|deviceName=sevtest.sevone.com|deviceIp=10.129.15.215|objectName=/var|indicator=dskAvail|type=GAUGE|pollTime=2019-01-08 14:20:11|pollValue=0.000000|pollTimestamp=1546957211.971767

    deviceId=117|deviceName=sevtest.sevone.com|deviceIp=10.129.15.215|objectName=/var|indicator=dskAvail|type=GAUGE|pollTime=2019-01-08 14:25:11|pollValue=0.000000|pollTimestamp=1546957511.782259

    Each line in the output above contains the following information.

    • deviceId - The device ID.

    • deviceName - The device name.

    • deviceIp - The IP address of the device.

    • objectName - The object name.

    • indicator - The indicator name.

    • type - The type of indicator.

    • pollTime - The date and time that the device was polled.

    • pollValue - The raw value that was returned.

    • pollTimestamp - The Unix timestamp indicating the time that the device was polled.

Tips

Accurate Configuration

Make sure to configure the .conf file (which you copy from the template file example-dataminer.conf) accurately in order to produce desired results. Double-check the names of device groups, object groups, etc. in SevOne NMS to confirm that the names match the names you are using in the configuration.

All or Nothing

If a problem occurs that prevents writing to the disk–for example, the disk is full–Data Miner won't provide partial results. It will only provide the data set as a whole.