Processes and Logs

Core Processes

• *MySQL Config Server - Synchronizes the Config databases of clustered appliances.

• *MySQL Data Server - Synchronizes the Data database of clustered environments.

• nginx - Displays the SevOne NMS web user interfaces.

• PHP-FPM - PHP handler used to run PHP scripts in the SevOne NMS GUI.

• SSH Daemon - Secure shell used for remote login to operate network services securely on an unsecured network.

• Syslog - Manages the logs that appear on the System Logs tab.

Optional Processes

• HTTP Proxy - Forwards VMware data among peers.

• sFlow Converter - Enables the NetFlow Collector process to receive sFlow data.

• Traffic samplicator (port 9997) - Listens on UDP port 9997 and replicates the traffic to any number of IP addresses and ports, similar to a broadcaster.

  By default, the samplicator is disabled. For the samplicator to persist as enabled across reboots and restarts of supervisord, using a text editor of your choice, edit /etc/supervisord.d/samplicator_9997.ini file.

  Edit /etc/supervisord.d/samplicator_9997.ini file to set 'autostart' to 'true'.

  $ vi /etc/supervisord.d/samplicator_9997.ini

  ...

  ...

  autostart=true

  ...

  ...

  After updating /etc/supervisord.d/samplicator_9997.ini, execute the following commands to update supervisorctl and to start/restart the service.

  $ supervisorctl reread

  $ supervisorctl update

  $ supervisorctl restart samplicator_9997

  For details on samplicator, please refer to section Samplicator below.

  When you execute the following command to perform the services check, samplicator service is ignored and the check does not inform the user whether or not the samplicator is running.

  $ SevOne-act check services

SevOne Daemons

• *SevOne Data Daemon - Inserts polled data into the database.

• SevOne Data Publisher - SevOne component that listens for new data points for devices and publishes them to Apache Kafka broker or Apache Pulsar broker cluster.

• SevOne FlowDB Daemon - Processes raw NetFlow data.

• SevOne Master/Slave (Leader/Follower) Monitor - Coordinates actions between the appliances in a Hot Standby Appliance peer pair

• SevOne Message Aggregator - Handles alert messages.

• *SevOne MIB Daemon - Caches MIB information and resolves OID numbers to text.

• SevOne NetFlow Collector - Handles the NetFlow poller.

• *SevOne Realtime Scheduler - Performs high frequency polling.

• *SevOne Request Daemon - Communicates with clustered appliances to collect data for reports.

• *SevOne Scheduler - Handles poll schedules.

• SevOne Stats - Processes data for self monitoring.

• *SevOne Trap Collector - Receives and processes SNMP traps.

• *SevOne xStats Dispatch Daemon - Performs name to ID resolution for xStats devices and associates xStats data to the correct peer.

• *SevOne xStats File Collector - Reads and parses xStats files.

• SevOne xStats Ingestion Resolver Daemon - Performs name to ID resolution for xStats objects and indicators.

SevOne Master/Slave (Leader/Follower) Actions

• Action: become leader - Makes the passive appliance in a Hot Standby Appliance peer pair take over and become the active appliance.

• Action: become follower - Makes the active appliance in a Hot Standby Appliance peer pair fail over and become the passive appliance.

• Action: format follower - Formats the database on the passive appliance in a Hot Standby Appliance peer pair.

SevOne Scripts

• SevOne Alert Mailer - Emails alerts that are configured to be emailed when the threshold is triggered.

• SevOne Device Mover - Processes and performs device moves from peer to peer that are initiated from the Device Mover.

• SevOne Discover Script - Handles all device discovery.

• SevOne Longterm Trim - Trims historical data based on the Cluster Manager setting.

• SevOne Report Mailer - Emails reports that are scheduled to be emailed.

• SevOne Shortterm Backup - Backs up short term data that is stored in memory.

SevOne Utilities

• SevOne Longterm Cacher - Caches and processes baseline data.

• SevOne Longterm Updater - Writes data stored in short term memory to the disk every two hours.

• SevOne Shortterm Trimmer - Trims short term data.

• SevOne Threshold Checker - Manages alerts.

System

• messages - Displays the generic log for all un-grouped messages.

• kern - Displays output of command /usr/bin/dmesg, which prints the kernel ring buffer.

Script Logs

• SevOne-backup-config-data.log

• SevOne-backup.log

• SevOne-checkmate.log

• SevOne-device-mover.log - Displays the log of the devices that you move between the peers in the cluster

• SevOne-devices-deletion-queue.log - Displays the log of the devices that are added to the deletion queue to be deleted.

• SevOne-ffupdater.log

• SevOne-generate-admin-messages.log

• SevOne-mib-synchronize.log

• SevOne-summary-table-tool.log

• SevOne-tablecacher.log

• SevOne-top-highpolld.log

• SevOne-top-polld.log

• aggregated-netflow-rollup.log - Displays the log that states when the daily data points for aggregated flow data ran

• alertmailer.log - Displays the log of the script that emails new alerts. Look here when alert emails are not sent or received

• cacher.log

• checksshd.log

• disableUsers.log

• discover-netflow.log

• discover-schedule.log

• discover-thereshold.log - This utility runs frequently throughout the day and sends new alerts to the messageswitch daemon

• discover.log

• highfreqpoller.update-by-time.log - Displays the log of the high frequency poller to inform you of the poll status

• ipmi-message.log

• mailreports.log - Displays the log of the script that emails reports. Look here when report emails are not sent or received

• mysql-replication-maintainer-config.log - Displays logs for MySQL config database

• mysql-replication-maintainer-data.log - Displays the logs for the MySQL data database

• mysqloptimize_config.log

• mysqloptimize_data.log

• namflow.log

• periodic.shortterm.backup.log - Displays the log for the short term backup. This utility writes the status of the periodic memory-table backups that are made when a server reboots

• proxy-write-config.log

• rapid-plugins-pdf.log

• rest-api-keepalive.log

• sevone-cert-update.log

• snmpd-restart.log

• sync-ldap-groups.log

• sysuptime-normalize.log

• trim-alerts.log

• trim-bulkdlogs.log

• trim-device.log

• trim-longterm.log

• trim-mysqllogs.log

• trim-netflow.log

• trim-netflowaggregate.log

• trim-rtagx.log

• trim-sessions.log

• trim-shortterm-netflow.log

• trim-shortterm.log

• trim-shorttermaggregate-daily.log

• trim-shorttermaggregate-hourly.log

• trim-shorttermaggregate-monthly.log

• trim-shorttermaggregate-sixhourly.log

• trim-shorttermaggregate-weekly.log

• trim-sixhourlynetflow.log

• trim-temporarytable.log

• trim-traps.log - Displays the log of the traps received. The output is in hexadecimal format

• updater.log - Displays the logs for the hourly, daily, weekly, monthly, quarterly, and yearly updater. This utility writes short term memory data to disk

• updateraggregate.daily.log

• updateraggregate.monthly.log

• updateraggregate.sixhourly.log

• updateraggregate.weekly.log

• upgrade-appliance.log

• vcenterupdate.log

• write-ldap-certs.log

Other Logs

• /var/date.log

General Logs

• SevOne-audit.log

• SevOne-clusterd.log

• SevOne-datad.log

• SevOne-device-scand.log

• ingestion/SevOne-dispatchd.log

• ingestion/SevOne-fcad.log

• SevOne-flowdb.log

• SevOne-highpolld.log

• ingestion/SevOne-ingestion-resolved.log

• SevOne-masterslaved.log - Displays information about the active appliance in a Hot Standby Appliance peer pair and its relationship with the passive appliance in the peer pair

• SevOne-mibd.log

• SevOne-netflow-cleanup.log - Displays the NetFlow daemon updater process

• SevOne-polld.log

• SevOne-requestd.log

• SevOne-searchd.log

• SevOne-statsd.log

• SevOne-topologyd.log

• SevOne-trapd.log - Displays the trap daemon

• cron.log

• discovery.log - Displays the log of the regular discovery script, which runs frequently throughout the day to discover device updates and new devices

• logrotate.log

• messageswitch.log - Displays the log of the alert handler. All new alerts come through this system

• mysql/mysqld.err

• mysql/mysql2.err

• mysql/mysqld_multi.log

• net-snmpd.log

• nginx.err

• nginx.log

• nginx/access.log

• nginx/error.log

• php-fpm.err

• php-fpm.log

• php-fpm/error.log

• php-fpm/www-error.log

• rest-api/SevOne-rest-api.log

• sdp.log

• sftp.log

• soa.log

• sshd.log

• tacc.log

• trim.log

• xStats-parsers.log

• xstats/ALU5620SAMTransform/adapter.log

• xstats/AwsTransform/adapter.log

• xstats/CanaryTransform/adapter.log

Configure Samplicator

1. Copy the samplicator example configuration file, /etc/conf.d/samplicator.example.confd, to /etc/conf.d/samplicator.1234.confd.
   

   Port 1234 is being used as an example here on which this instance of the samplicator service is being configured for and will listen on.

   $ cp /etc/conf.d/samplicator.example.confd /etc/conf.d/samplicator.1234.confd

2. Using a text edit of your choice, edit /etc/conf.d/samplicator.1234.confd file to add device IP address 10.0.0.61 and save it.

   Example

   $ vi /etc/conf.d/samplicator.1234.confd

    

   #Config file format:

   #a.b.c.d[/e.f.g.h]: receiver ...

   #where:

   #a.b.c.d is the sender's IP address

   #e.f.g.h is a mask to apply to the sender (default 255.255.255.255)

   #receiver see above.

   10.0.0.61: 10.0.0.60/9996

   /etc/conf.d/samplicator.1234.confd file format...

   The first column is where the samplicator expects the UDP packets from. i.e., 10.0.0.61, as shown in the example above.

   The second column is where the UDP packets go to followed by the port number. i.e., 10.0.0.60/9996, as shown in the example above.

   Alternatively, you may use 0.0.0.0 as the receiver IP address if it does not matter where the data is coming from and you want to forward all UDP traffic coming in on the specified port.

3. Now that the configuration file has been modified, set the parameters for supervisord to start the samplicator for this specific port forwarding request.

   Samplicator configurations can be created individually, or many different incoming / outgoing pairs can configured for the same samplicator instance. This depends on the requirements of the environment.

4. Copy /etc/supervisord.d/samplicator_9997.ini file to /etc/supervisord.d.<master or dnc or slave folder>/samplicator_1234.ini. Please see the note below to determine whether to copy the file in master or dnc or slave folder.
   

   Master / Leader or any other role...

   If the samplicator service needs to be running when the appliance is in Master / Leader state, copy the file to /etc/supervisord.d.master folder.

   For an active DNC appliance, copy the file to /etc/supervisord.d.dnc folder.

   In any other role, copy to /etc/supervisord.d.slave folder.

   For the steps below, we will assume that the appliance is in Master / Leader state.

   Example

   $ cp /etc/supervisord.d/samplicator_9997.ini /etc/supervisord.d.master/samplicator_1234.ini

5. Using a text edit of your choice, edit /etc/supervisord.d.master/samplicator_1234.ini file to update the program name to reflect the samplicator port being used. i.e., [program:samplicator_1234]. Also, update the command to reflect the configuration file name (using option -c) and the port (using option -p). After the updates, save the file.
   

   Example

   $ vi /etc/supervisord.d.master/samplicator.1234.ini

    

   [program:samplicator_1234]

   command=/usr/bin/samplicate -S -c /etc/conf.d/samplicator.1234.confd -p 1234 -d0

   stdout_logfile=/var/log/samplicator.log

   stderr_logfile=/var/log/samplicator.err

   priority=500

   autostart=true

   startsecs=10

   startretries=10000

   autorestart=true

   You may leave stdout_logfile and stderr_logfile as-is if you want to send the output to the same log file as the other samplicator instances. Or, you may also choose to give it a separate log file name.

6. After configuring /etc/conf.d/samplicator.1234.confd and /etc/supervisord.d.master/samplicator_1234.ini files, execute the following command.
   

   $ supervisorctl reread && supervisorctl update

    

   samplicator_1234: available

   samplicator_9997: changed

   samplicator_9997: stopped

   samplicator_9997: updated process group

   samplicator_1234: added process group

7. Start the samplicator process for the new configuration.
   

   $ supervisorctl start samplicator_1234

Test Samplicator

You may now run tcpdump on the destination port you have configured above to ensure that the samplicator is working properly. If you do not have any data coming in, you may send the test data to the samplicator port on the appliance.

Some Considerations

Ensure that the port you have chosen is not already in-use for the samplicator to listen on. If another process is bound to the port, the samplicator will fail to start.