Cheshire Cat Computing

Folks,

I've been trying to make a standalone version of trend.cgi
and am quite close, I believe. But reading about the RRDs:functions
on the man page hasn't gotten me to this point.

When I run, the error "Could not read in rrd file" comes in.

Please have a look at this code and let me know what you think.

My goal is to use this to predict the x-axis value for a specific
y-axis value and then determine if that x-axis value is in the
nearterm and hence be predictive numerically as well as
graphically, to drive other tools for the less volatile measurements.

The problem below seems to be my lack of understanding of
what constitutes the argument to the RRDs last function

Thanks,

Stuart

#!/usr/local/bin/perl -w

use strict;
use CGI;
use FileHandle;
use Text::ParseWords;
use File::Basename;
use RRDs;

# Variables
our ($rrdfile ) = '';
our( $DECAY ) = 1;
our( $device, $community, $targetwindow, $target, $file, $backurl )
= ( "","public","graph","","","");
our( $conffile ) = "/usr/local/apache/htdocs/mrtg/routers2.conf"; # overridden by passed parameters
our( $routersurl ) = '';
our( $q ) = new CGI;
our( %headeropts ) = ();
our( $thishost ) = $q->url();
our( %config ) = ();
our( $authuser ) = '';
our( $pathsep ) = '/';
our( %target ) = ();
our( $tempfile ) = '/tmp/trend.foo';
our( $ds, $rrddata , $starttime, $endtime, $interval) = ('','','','','');
our( $trendstart, $trenddelta ); # array references
our( $dwmy ) = 'y';
our( %interfaces );
our( $workdir ) = '';
our( $monthlylabel ) = '%W';
our( $dailylabel ) = '%H';

#######################################################################
# calculate weighted SD for given start/delta
# weighted sd = square root of weighted average of squares of difference
sub wsd($$$) {
my( $idx, $start, $delta ) = @_;
my( $c, $tot, $w, $row, $v ) = (0,0,0,0,0);
my( $fx );

$fx = $start;
$w = 1;
foreach $row ( 0 .. $#$rrddata ) {
$v = $rrddata->[$#$rrddata - $row]->[$idx];
if($v) { $tot += ($v-$fx)*($v-$fx) * $w; $c += $w; }
$w *= $DECAY; # Next value is less significant
$fx -= $delta;
}
if($c) {
$tot = sqrt($tot / $c);
} elsif( $tot ) {
$tot = $tot * $tot; #XXXXXXX temp
}

return $tot;
}
#######################################################################
# This function will retrieve the RRD data, calculate the value and
# slope of the trend line, then call the outputxml function and rrdtool
# to create XML and the RRDTool database.
sub do_trending {
my( $rrdfile ) = @_;
my( $timeframe ) = "1d";
my( $answer, $status, $e );
my( $n, $idx, $c, $tot, $w, $row, $v, $deltadelta );
my( $sd, $delta, $sda, $sdb, $lim );

print "Starting trending function - rrdfile = $rrdfile\n";

# set intervals (in seconds): 5min, 30min, 2hr, 1day
foreach ( $dwmy ) {
if( /w/ ) { $timeframe = "200h"; $interval = 1800; last; }
if( /m/ ) { $timeframe = "800h"; $interval = 7200; last; }
if( /y/ ) { $timeframe = "400d"; $interval = 3600*24; last; }
$timeframe = "2000m"; $interval = 300;
}

# Read in the RRD file
$endtime = RRDs::last($rrdfile);
$e = RRDs::error;
die "Could not read in rrd file\n" if ($e);
( $starttime, $interval, $ds, $rrddata )
= RRDs::fetch($rrdfile,"AVERAGE","-r",$interval,
"-s",($endtime-(400*$interval)),"-e",$endtime);
$e = RRDs::error;
die "Could not fetch data from rrd file\n";

# Calculate trends
# this results in (a) last update time, (b) avg value then,
# (c) change in value per interval
$trendstart = [ 0,0 ];
$trenddelta = [ 0,0 ];
foreach $idx ( 0, 1 ) {
# calculate weighted average (start point for line)
$w = 1; # weight of current sample
$tot = 0; # total of values
$c = 0; # total weight
$n = 0; # count of samples processed
foreach $row ( 0 .. $#$rrddata ) {
$v = $rrddata->[$#$rrddata - $row]->[$idx];
if($v) { $tot += $v * $w; $c += $w; $n++; }
$w *= $DECAY; # Next value is less significant
}
$trendstart->[$idx] = $tot / $c if($c);
errlog("Weighted average for $idx is: ".$trendstart->[$idx]." = $tot / $c, $n valid samples");

# now calculate best angle of line.
# vary delta as weighted standard deviation decreases, until
# we dont get a gain by going up or down.
$delta = $trenddelta->[$idx];
$deltadelta = $trendstart->[$idx]/100.0;
$lim = $deltadelta/1000.0;
$sd = wsd($idx,$trendstart->[$idx],$delta);
$n = 0;
while(($n < 100)and ($deltadelta>$lim)) { # put a cap on iterations
$n++;
errlog("Delta=$delta, Deviation=$sd : deltadelta=$deltadelta, limit $lim");
$sda = wsd($idx,$trendstart->[$idx],$delta+$deltadelta);
$sdb = wsd($idx,$trendstart->[$idx],$delta-$deltadelta);
errlog("up->$sda, down->$sdb");
if($sd<$sda and $sd<$sdb) { # we are in a trough
$deltadelta /= 2;
} elsif( $sda < $sdb ) {
$delta+=$deltadelta;
$sd = $sda;
} else {
$delta-=$deltadelta;
# chage direction
$deltadelta = -($deltadelta/2);
$sd = $sdb;
}
}
$trenddelta->[$idx] = $delta;
}

}

$tempfile = "/tmp/trenddata";

$rrdfile = "/usr/local/apache/htdocs/mrtg/aqaba.idc.vzwcorp.com.disk.rrd";
$rrdfile = "aqaba.idc.vzwcorp.com.disk";
$rrdfile = "aqaba.idc.vzwcorp.com.disk.rrd";

do_trending($rrdfile,"$tempfile.rrd");

If by 'standalone trend.cgi' you mean one that doesn't need the rrdtool executable and is pure perl, then the version currently shipping with routers.cgi v2.15beta matches this description. It works under windows and unix, too.

I'm hoping to be able to add multiple do_trending functions to the trend.cgi so that you can pick which method to use... curently, I've only added a second method of calculating the basepoint for the best-fit line (either use weighted average, or last value)

_________________
Steve Shipway
UNIX Systems, ITSS, University of Auckland, NZ
Woe unto them that rise up early in the morning... -- Isaiah 5:11

No problem -- since posting the above, I was able to dig around
and construct what I needed. I can pull the time for any given
y-value, compare it to time(), see if that is nearterm, and
then alarm on it.

What I did was take the create_xml and made it a do_prediction
function. Instead of creating an xml file it takes an y-value as
parameter and returns the x-value at which the y-value has
been surpassed.

The caller then can compare the returned value with time(),
see if it is greater but not by too much, and output its prediction
of when the threshold will be reached.

The point is to avoid people having to look at graphs too much
for prediction if they don't want to. Also, this can link into our
paging system or an emailed report system. I.e. the benefits of
the trending capability of routers2.cgi can be parlayed to other
areas for other benefits. For things that aren't too volatile naturally.

Stuart