Add/Update (OTADD)

Otadd is the executable called from the IBM NetView OverTime menu when an “Add/Update” is requested. It is passed the list of currently selected objects from the OpenView map/s. If no objects are selected, the menu item will not be available. Each object is then SNMP queried for additional information not available within the IBM NetView database. Otadd firstly resolves the SNMP SysName and selects an IP address from each device to use to communicate with that device. OverTime determines which IP address OpenView will use to communicate with the device and uses this address. The IBM NetView snmpCol.conf file is then searched to see what, if any, collections may already be in place for this device. Two classes of collection are considered. I/O collections involve ifInOctets, ifOutOctets, ifInErrors, ifOutErrors, CiscoAvgBusy5 and sysuptime are in the I/O class. All other collections are considered as special user defined collections. If no collections exist then otadd will setup collections based purely on I/O. If a user defined collection is setup and no I/O collection is active, then only the user-defined collection will be setup within OverTime. If both I/O and user defined collections are setup, then both will be setup within OverTime. OverTime will work with any numeric collection including EXPRESSION collections.

For I/O collections (from routers and switches)

1 instance = 1 interface's InOctets, OutOctets, InErrors and OutErrors.

1 instance = Cisco CPU Average Busy 5

For all other collections 1 instance = 1 SNMP instance.

For I/O collections, the device is queried for all available interfaces that meet certain criteria. I.E. the interface must be administratively up, not be any of: a ds1, a loopback, a Null, a Dialer, a CPU and have a reasonable interface speed >0 and <400Mb/sec. The acceptable interfaces are then queried for their speed, their names setup in the device, their maximum bandwidth and any associated IP address all of which then automatically appear in the OverTime master configuration file for this device and will ultimately appear on the web page and its graphs.

If the master configuration file for the device already existed and the legends, y axis label or graph title were changed for user-defined collections, then these changes are preserved by otadd.

Otadd writes an snmpCollect merge file that is then merged with the existing collections so that IBM NetView will perform the collections.

Otadd also writes the master configuration file in OverTime/cfgs. This file’s name is the device’s SNMP sysName (or its IP address if no name is available either of which will be referred to as sysName within this document) with an extension of “.mst”. (See above for the definition of this file).

Otadd then updates overtime.csv (see later) and the master HTML file, index.html.

Otadd creates the directories for the RRD databases to be stored in OverTime/data/ under the devices sysName directory.

Otadd creates the directories for the graphics to be held in the web server’s area OverTime/ under the devices sysName directory.

Otadd sets up the RRDs based on what it finds and other user setting within overtime.cfg.

User-defined collections cause otadd to create RRDs based on the collection name. These databases are also configured based on the collection frequency setup when the RRD is initially created. IT IS NOT CHANGED while the RRD database exists. The RRDs are given names that commence with sysName and then followed by the SNMP interface number for I/O collections. For user-defined collections, the RRD is called sysName and then the MIB name (as found in snmpCol.conf) followed by the instance number discovered by “snmpwalking” the device for this OID.

Otadd checks to see if the collections being requested will alter the work being performed by snmpCollect and only if it considers there is a change will it request snmpCollect to re-initialise. When snmpCollect re-intialises it takes a little time for the collections to re-commence.

Cisco devices are special cased and additional information is extracted and made available for them such as CiscoAvgBusy5 for CPU utilization when available.

Sensor Probes/Sensor Hubs are also special cased. Their humidity and temperature sensors are intelligently setup including the names of the sensors and their critical limits.

External/Non-SNMP Collections

OverTime can be used to record and display collections gathered by your own application or script and not just by NetView's snmpCollect daemon. These types of collections are termed "external or non-SNMP collections" in OverTime. You use non-SNMP collections to monitor from your server any kind of statistical value. Maybe you want to monitor your web server’s response time or the number of cars going past your window. If your server can count them, then OverTime can help you record them. This is what a graph of web server response times looks like:

Setting Up a new TCP port Collection.

An example of this type of collection will help to make the process clear. Let's assume that you would like to monitor the response times for loading the main web page from two web sites (www.myco.com and www.myotherco.com) and you want to see the results on your OverTime page under the label ‘Web_Responses”.

You can use this check_port.pl script as a starting point.

1) Copy the check_port.pl script and give it a name like http_time.pl on your OverTime server.

2) Edit the new script and change the settings at the top. The settings that you need to modify are the TCP port number, the string to send when the port opens and the string that you expect to be returned if the test is successful. Leaving the ‘testing’ value set to 1 will help you test the process. For our example, we need to make these changes:
$port=80; # 80 for http - User chooses what they want!
$send = "GET index.html\n"; # What do you want sent
$expect="\<\/html\>"; # What you expect to be returned if it worked.
$testing=1; # 1 if we're testing, 0 if we're really collecting
$threshold = 0.5 # A threshold of 0.5 seconds to respond
$runcmd = "echo Threshold breached."; # Command to run when threshold is breached
$script_name = "http_time"; # The name of this script

3) Test your new script with a command like ‘http_time.pl Web-Responses www.myco.com www.myotherco.com”

4) When you have this working correctly, remember to edit the script and set $testing=0

5) To add your new collection to OverTime you run the otadd command:
“otadd --external G /var/opt/OverTime/cfgs/http_time.pl Web_Responses www.myco.com www.myotherco.com”.
This instructs OverTime to create the appropriate configuration files and also update the PingTime-xxx command file.
The “--external” tells otadd to create an external/non-SNMP collection configuration.
The “G” indicates if the numbers collected are a gauge. Alternatively, it could have been a "C" for counter. Counters are for values that continuously increase like the count of packets through an interface. Gauge is for values that are absolute like the current temperature or the current response time.
The "/var/opt/OverTime/cfgs/http_time.pl" (or on Windows "c:\OverTime\bin\http_time.pl") is the command, script or executable to run, including its full path.
“Web_Responses” is the name that will be used on the main OverTime HTML page and must also be a valid directory name. It may be a new name or the name of an existing OverTime device.
“www.myco.com www.myotherco.com” is the list of targets to be tested.

6) Once this has completed, you will be able to browse to results via the OverTime index.html page. (Remember, you won't see any results until the first collection has completed)

If "sysName" does not already appear in the index.html page for OverTime, then it will be added by otadd.

RRD datafiles will are created and the configuration of the regularly run (via 'at' or 'cron') command "PingTime-300" (see PingTime) is modified.

When PingTime-300 is next run, it then calls this command:

“/var/opt/OverTime/cfgs/http_time.pl Web_Responses www.myco.com www.myotherco.com”.

You can download the sample Perl script, check_port.pl and it looks like this:

__________________________________________________________________________________________

#!/usr/local/bin/perl

# Web server checking looks like this:

$port=80; # 80 for http - User chooses what they want!

$send = "GET /index.html\n"; # What do you want sent

$expect="\<\/html\>"; # What you expect to be returned if it worked.

$testing=1; # 1 if we are testing , 0 if we're really collecting

$threshold = 0.5; # A threshold of 0.5 seconds to respond

$runcmd = "echo Threshold breached."; # Command to run when threshold is breached

$script_name = "http_time"; # The name of this script

######################## Probably won't need to change anything below here ##################

use IO::Socket;

use Time::HiRes qw(gettimeofday);

for($ind=1; $ind< @ARGV; ++$ind){

$resptime = "U";

# Start the timer here!

# Open the socket

$start = now();

$soket = IO::Socket::INET->new(Proto => "tcp", PeerAddr => $ARGV[$ind] ,PeerPort => $port, Timeout => 2, );

$resptime="U"; # Failed by default

if ( $! eq '' ) { # Everything OK so far

$soket->autoflush(1);

# Send some protocol dependant data to the server

print $soket "$send"; # Send something to the port for it to respond

# Read back the results

while(<$soket>){

if($testing > 0){ print; }#Print all the responses if testing is set greater than 1

# Check for something in the result?

if(/$expect/) {

$resptime = now() - $start;

if ($testing) { print "Expected response of '$expect' was seen in $resptime seconds for $ARGV[$ind]\n"; }

} # Success

}

close $soket;

} else {

# Tell someone we've had a problem

print ("Port $port to $ARGV[$ind] did not open\n");

}

# End the timer here

# if $resptime=="U" then the time is not valid so don't check the thresholds.

# If the threshold has been breached, run a command!

if($resptime ne "U") {

if($resptime > $threshold) {`$runcmd`; }

} elsif ($testing) { print "Expected response of '$expect' was NOT seen for $ARGV[$ind]\n"; }

if ($testing > 0) {

`"/opt/OV/bin/devupd"`; # Just run an update with no arguments - should see the usage

} else {

`"/opt/OV/bin/devupd" $ARGV[0] $script_name $ARGV[$ind] Now $resptime`;

}

}

sub now {

my(@tmp) = gettimeofday();

return $tmp[0] + ($tmp[1] / 1000000.0);

}

__________________________________________________________________________________________