newspaint

Documenting Problems That Were Difficult To Find The Answer To

Monthly Archives: Sep 2013

Cacti Running Out Of Memory In adodb.inc.php

I upgraded Debian to Wheezy testing and suddenly Cacti stopped working.

In my Apache logs I found the following error:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 65484 bytes) in /var/www/html/cacti/cacti-0.8.7g/lib/adodb/adodb.inc.php on line 833

I patched adodb.inc.php to dump the SQL it was trying to run, and I cut and paste one of the statements into a mysql session:

MariaDB [cacti]> select value from `cacti`.`settings` where name='log_verbosity';
ERROR 126 (HY000): Incorrect key file for table './cacti/settings.MYI'; try to repair it
MariaDB [cacti]> quit

So – it looked liked the tables needed repairing (maybe the upgrade from MySQL to MariaDB broke the tables). From the command line I ran the following command to repair all my tables in all my databases:

mysqlcheck -u root -p --auto-repair --all-databases

This solved the problem. Cacti works again for me.

PHP Suhosin Not Supported in Debian Wheezy

I performed a dist-upgrade of my Debian testing install to Wheezy recently.

I began to see, on the console, messages like:

Killed process 6745 (php) total-vm:1368712kB, anon-rss:772304kB, file-rss:0kB

I tried to run PHP from the command line and got the following error message:

PHP Startup: Unable to load dynamic library ‘/usr/lib/php5/20100525+lfs/suhosin.so’ – /usr/lib/php5/20100525+lfs/suhosin.so: cannot open shared object file: No such file or directory in Unknown on line 0

Apparently suhosin was removed from Wheezy testing.

The recommended action from the upgrade notes is to run:

dpkg --purge php5-suhosin

… which will remove the leftover configuration in /etc/php5/conf.d/suhosin.ini.

A Basic Ant Build Script For Simple Java Projects

Here is a basic Apache Ant build script I use for taking one or more source Java files and associated jars and bundling them all into a single jar with a main class.

I have the following directory structure:

|--build.xml
|--src\
|  +--MyMainClass.java
|  +--OtherClass.java
+--lib\
   +--dependency.jar

When I run ant jar using the below build.xml it will then create new directories:

|--build
|  +--classes
|     +--MyMainClass.class
|     +--OtherClass.class
+--jar
   +--myproject.jar

Running ant clean will remove the newly created folders build/ and jar/.

This tutorial is also very helpful.

<project name="myproject">
  <property name="src.dir"     value="src"/>

  <property name="build.dir"   value="build"/>
  <property name="classes.dir" value="${build.dir}/classes"/>
  <property name="jar.dir"     value="jar"/>
  <property name="jar.file"    value="${jar.dir}/${ant.project.name}.jar" />

  <property name="lib.dir"     value="lib" />

  <property name="main-class"  value="MyMainClass" />

  <path id="classpath">
    <fileset dir="${lib.dir}" includes="**/*.jar" />
  </path>


  <target name="clean">
    <delete dir="${build.dir}" />
    <delete dir="${jar.dir}" />
  </target>

  <target name="compile">
    <mkdir dir="${classes.dir}" />
    <javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="classpath" />
  </target>

  <target name="jar" depends="compile">
    <mkdir dir="${jar.dir}" />
    <jar destfile="${jar.file}" basedir="${build.dir}">
      <manifest>
        <attribute name="Main-Class" value="${main-class}" />
      </manifest>
      <fileset dir="${classes.dir}" includes="**/*.class" />
      <fileset dir="${src.dir}" includes="**/*.java" />
      <zipgroupfileset dir="${lib.dir}" includes="**/*.jar" />
    </jar>
  </target>
</project>

If I want debugging enabled (see also Ant javac parameters for more options):

  <target name="compile">
    <mkdir dir="${classes.dir}" />
    <javac srcdir="${src.dir}" destdir="${classes.dir}" classpathref="classpath" debug="true" />
  </target>

Compiling CORBA IDL Files As Dissectors Into Wireshark

Download a version of Wireshark no earlier than 1.8 (at the time of writing the most recent was 1.10.2, note that I’ve had a terrible time trying to get Wireshark 1.10.x to compile my IDLs, it complains about strings in the IDL files, and have been forced to remain on version 1.8).

Unpack Wireshark

Download a Wireshark tarball from the Wireshark Download site and place it in /tmp/.

export WIRESHARKVER=1.8.10
export ROOTDIR=/tmp
cd $ROOTDIR
tar -xvjf wireshark-${WIRESHARKVER}.tar.bz2

Prepare IDL Files

Create a directory to put your interface definition language (IDL, or *.idl) files into (myidl/). You’ll also create a directory to receive the compiled IDL files (*.idl.c) named mybuild/.

cd $ROOTDIR
mkdir myidl
mkdir mybuild
cd myidl

Copy your *.idl into $ROOTDIR/myidl/.

Create Wireshark Dissector Sources from IDL Files

cd $ROOTDIR/myidl
for i in *.idl; do
  export FULLPATH=`pwd`/$i
  export TOOLDIR=$ROOTDIR/wireshark-${WIRESHARKVER}/tools
  pushd $TOOLDIR
  ./idl2wrs $FULLPATH > $ROOTDIR/mybuild/$i.c
  popd
done

If you are missing the omniidl utility then you may have to install a package that supplies omniORB. Note that if your Python complains about the wrong ELF class then just rename the offending library file’s extension and Python should then go looking for the 64-bit equivalent (applies to RedHat distributions).

Add New Dissectors to Wireshark Build

cp $ROOTDIR/mybuild/*.c $ROOTDIR/wireshark-${WIRESHARKVER}/epan/dissectors/
cd $ROOTDIR/wireshark-${WIRESHARKVER}/epan/dissectors/

ls -1 *.idl.c # cut and paste into next step

vi Custom.common
# add copied list of *.idl.c into CUSTOM_DISSECTOR_SRC = with trailing backslashes

Now while editing Custom.common add the names of the compiled dissector files as the following example shows (no trailing backslash on the last line or you’ll get compilation errors):

CUSTOM_DISSECTOR_SRC = Connection.idl.c \
Session.idl.c \
User.idl.c

Save the file and continue:

cd $ROOTDIR/wireshark-${WIRESHARKVER}
./autogen.sh # you may have to install automake
./configure --prefix=/usr/local/myshark
nice make -j 2 # -j 2 means use 2 threads (not necessary)
make install

Building a Static TShark on RedHat 5.9 (Tikanga)

RedHat 5.9 uses many outdated packages. Unfortunately Wireshark versions from at least v1.8.2 (and possibly earlier but at least after v1.6.16) require glib 2.14 or better when RHEL5 only provides glib 2.12 (RPM glib2-2.12.3-4.el5_3.1).

Initially I tried downloading the latest glib from http://www.gtk.org/download/linux.php which took me to glib v2.36. However I quickly discovered that this required Python 2.5 which, as you guessed, was more recent than the Python 2.4 supplied with RedHat.

I downloaded glib v2.26 and extracted it and built it with the following configuration line:

./configure --enable-static --prefix=/tmp/glib
make
make install

Next I built with the following configure line for Wireshark after downloading v1.10.2 and extracting:

CFLAGS=-fPIC LD_LIBRARY_PATH=/tmp/glib/lib PKG_CONFIG_PATH=/tmp/glib/lib/pkgconfig \
  ./configure \
    --prefix=/usr/local/myshark \
    --disable-wireshark \
    --enable-static=yes \
    --enable-shared=no
make
make install

The CFLAGS=-fPIC was necessary to complete a static build. This forum post documented how to specify a custom glib directory when building tshark (by specifying the LD_LIBRARY_PATH and PKG_CONFIG_PATH environment variables).

Millisecond Time Resolution X/Y Scatter Plotting With Gnuplot

So you have timestamps with a resolution of milliseconds and you want to plot them on a X/Y scatter graph using Gnuplot.

This is possible because the %S specifier in timefmt accepts a double (i.e. a floating point value) so that the fractional portion of the time will be interpreted as a fraction of seconds.

You can print fractional seconds on the X-axis using the set format x “%H:%M:%.3S” command.

Here is a Gnuplot plot file to demonstrate this:

set terminal png size 400,320 font 'Verdana,8'
set output 'test.png'
set datafile separator ","
set xdata time
set timefmt "%Y-%m-%d %H:%M:%S"
set format x "%H:%M:%.3S"
set title 'Speed over Time'
set xlabel "Timestamp"
set ylabel "km/h"
set xrange ["2013-09-16 10:59:05.000":"2013-09-16 10:59:06.000"]
set yrange [0:12]
set xtics out rotate by -80
set key below
set pointsize 1.0

plot '-' using 1:2 title 'My Car' with points pointtype 2
        2013-09-16 10:59:05.040,5
        2013-09-16 10:59:05.240,7
        2013-09-16 10:59:05.470,10
        2013-09-16 10:59:05.780,3
        e

If you run this you should get the following image created:

Gnuplot 4.6 Millisecond Resolution Scatter Plot

Gnuplot 4.6 Millisecond Resolution Scatter Plot

The documentation for Gnuplot is available at the Gnuplot documentation home page. Note that you might need a recent version of Gnuplot to achieve millisecond-resolution plotting as it appears this was introduced into version 4.6.

Why Google’s Go Language Is Not Ready For The Enterprise

So a few times I’ve been tempted to “try out” Google’s language named “Go” (which actually is harder to search for on the Internet than the letter “C” because the word “go” is commonly used in expressive English whereas the letter “C” is rarely used in conversation).

However, just like Java, I keep hitting brick walls that prevent any practical use of the language in business.

In general Go appears has been engineered to satisfy a particular clique of developers with rose-coloured glasses on. While it could have been a significant boost to multi-processor application development it has adopted a rigidity around the semantics that will likely alienate a great majority who might have otherwise considered it.

Stubbornness isn’t new in a product that an author wishes to be used in a particular way. The author of FileZilla stubbornly resisted adding FTP proxy support to his product arguing that routers’ NAT should be correctly configured instead; but relented in version 3.0.4 RC1 / 3.0.3 / 2.2.32 after overwhelming demand for the feature. Unfortunately Perl, which used to be a very flexible language, recently started rejecting a commonly used qw operator idiom with no means of turning off the warning/error generated by its use – breaking generations of code out there in production today – a most unprofessional result of stubbornness.

Go Will Not Compile If A Variable or Import Is Unused

According to the language FAQ:

Some have asked for a compiler option to turn those checks off or at least reduce them to warnings. Such an option has not been added, though, because compiler options should not affect the semantics of the language and because the Go compiler does not report warnings, only errors that prevent compilation.

There are two reasons for having no warnings. First, if it’s worth complaining about, it’s worth fixing in the code. (And if it’s not worth fixing, it’s not worth mentioning.) Second, having the compiler generate warnings encourages the implementation to warn about weak cases that can make compilation noisy, masking real errors that should be fixed.

Whilst I absolutely agree that production code should not generate any warnings there are advantages to their presence.

  • sometimes languages change but old code needs to be run in a new environment – Perl, for example, recently changed the semantics of the qw operator that has broken a lot of old code; the inflexibility of the language to suppress or merely generate warnings for code that was legitimate is a rigidity that prevents pragmatic use in a workplace over time
  • the development cycle necessarily has broken code because it isn’t finished yet – during a bug fixing cycle one may comment out blocks of code to try and isolate a problem – but being unable to leave unused variables around severely impacts on the ability of this tried-and-tested technique of problem isolation to be implemented
  • a warning is something that is serious enough to know about – but, in the right circumstances – can be ignored; ever crossed the road when the lights were red but you looked both ways and no cars were coming? Consider yourself to be ignoring a warning. It happens every day in real lives. It happens in business, too. Nobody should be encouraged for recklessly ignoring warnings – but they are useful informational tools to those who make informed judgements off the back of them

It clearly makes no sense to prevent a program from compilation with a superfluous variable or import. No sense whatsoever. It makes sense to warn, just as C and C++ will, but no professional language would stop a developer from developing in this circumstance.

Exports Must Have a Leading Upper-Case Letter

In one of the more ridiculous restrictions of the language come the requirement to differentiate between methods and variables local to a module from those that are exported by mere capitalisation of the identifier.

To prove how ridiculous this is we have a FAQ questioning how a language without case (such as Japanese) can possibly export methods! In fact the sheer arrogance on display is beyond all reasonable sense with this quote from the FAQ: “The case-for-visibility rule is unlikely to change however; it’s one of our favorite features of Go”. Just enormously stupid. Forcing an atypical grammar to suit a personal itch.

Such inflexibility doesn’t leave any room for the developer to find their own style of naming conventions. And it requires that one innately knows the rules for naming conventions.

To the developer this makes experimentation/design nearly impossible as it means a search-and-replace across a file (to change the name of the variable right throughout the code) if one decides to change the export status of a method or variable rather than a simple public/private type keyword or an explicit export declaration.

Creating a Filled Stack Graph in GNUPlot

So you want to create a stack graph in gnuplot but there’s no “stack” function. This isn’t the first blog to be written about it but will document my approach to solving this problem.

Essentially you want to create several graphs drawn on top of each other. In this case if we have, say, 5 lines we want to stack on top of each other then we start with the first column and sum it with all the columns to the right of it and plot it with fillcurves (gnuplot will automatically choose a colour for us). Then we take the second column, sum it with all the columns to the right of it, and plot it with fillcurves which will be drawn over the top of the most recent plot. And so forth up to the last column of data which doesn’t need to be summed with anything else but plotted over the top of the others with fillcurves.

Here is an example of the essential plot commands (for an X/Y plot by date):

plot \
  "feeds.data" using 1:($2+$3+$4+$5+$6) title 'Bloomberg' with filledcurves x1, \
  "feeds.data" using 1:($3+$4+$5+$6) title 'Yahoo' with filledcurves x1, \
  "feeds.data" using 1:($4+$5+$6) title 'MSN' with filledcurves x1, \
  "feeds.data" using 1:($5+$6) title 'Google' with filledcurves x1, \
  "feeds.data" using 1:($6) title 'DowJones' with filledcurves x1

If you’re wondering what x1 is doing after filledcurves it specifies from which axis to fill the curve up to (x1 being the bottom horizontal axis).

If we were to populate feeds.data with:

2013-09-01,3,5,12,14,3
2013-09-02,5,6,12,15,6
2013-09-03,2,4,16,12,4
2013-09-04,1,3,18,11,3
2013-09-05,3,4,15,13,6
2013-09-06,7,3,12,21,9
2013-09-07,5,4,11,18,8

and fed it into the following plot:

set title "Articles Imported"
set xdata time
set timefmt "%Y-%m-%d"
set datafile separator ","
set terminal png size 480,400 enhanced truecolor font 'Verdana,9'
set output "articles_imported.png"
set ylabel "Articles"
set xlabel "Date"
set xrange ["2013-09-01":"2013-09-07"]
set yrange [0:10<*]
set pointsize 0.8
set format x "%d/%m"
set border 11
set xtics out
set tics front
set key below
plot \
  "feeds.data" using 1:($2+$3+$4+$5+$6) title 'Bloomberg' with filledcurves x1, \
  "feeds.data" using 1:($3+$4+$5+$6) title 'Yahoo' with filledcurves x1, \
  "feeds.data" using 1:($4+$5+$6) title 'MSN' with filledcurves x1, \
  "feeds.data" using 1:($5+$6) title 'Google' with filledcurves x1, \
  "feeds.data" using 1:($6) title 'DowJones' with filledcurves x1

The result is the following graph:

GNUPlot Graph Simulating a Stack Plot

GNUPlot Graph Simulating a Stack Plot


Using the Sum Function

I was recently made aware of this forum post which referenced this blog article – and shows a modernised and improved way of drawing a stack graph.

As of Gnuplot v4.4 the iterative plot for command can be used to generate the multiple graphs.

As of Gnuplot v4.6 the sum command can be used.

Putting this altogether let’s re-use our data file – but this time put the titles of the columns in the first row of the data:

,Bloomberg,Yahoo,MSN,Google,DowJones
2013-09-01,3,5,12,14,3
2013-09-02,5,6,12,15,6
2013-09-03,2,4,16,12,4
2013-09-04,1,3,18,11,3
2013-09-05,3,4,15,13,6
2013-09-06,7,3,12,21,9
2013-09-07,5,4,11,18,8

Now we replace the previous plot command with:

set title "Articles Imported"
set xdata time
set timefmt "%Y-%m-%d"
set datafile separator ","
set terminal png size 480,400 enhanced truecolor font 'Verdana,9'
set output "articles_imported.png"
set ylabel "Articles"
set xlabel "Date"
set xrange ["2013-09-01":"2013-09-07"]
set yrange [0:10<*]
set pointsize 0.8
set format x "%d/%m"
set border 11
set xtics out
set tics front
set key below
plot \
  for [i=2:6:1] \
    "feeds.data" using 1:(sum [col=i:6] column(col)) \
      title columnheader(i) \
      with filledcurves x1

The for [i=2:6:1] sets up a counter that increments variable i by 1 starting at 2 until it gets to 6 (note that the first data column is 2 because the date is in column 1).

The sum [col=i:6] column(col) command sums up the columns from column i to column 6.

The title columnheader(i) takes the title of the particular row from the header.

In pseudo-code (Perl) this looks like:

for ( $i = 2; $i <= 6; $i += 1 ) {
  my @to_sum;
  for ( $col = $i; $col <= 6; $col += 1 ) {
    push( @to_sum, value_in_column( $col ) );
  }

  plot( $i, sum( @to_sum ) ); # plot line $i
}

The resulting image is identical to the one above using the old-style plot command.

If our stack is not too complicated we can add black lines between each colour by adding to the plot command:

plot \
  for [i=2:6:1] \
    "feeds.data" using 1:(sum [col=i:6] column(col)) \
      title columnheader(i) \
      with filledcurves x1, \
  for [i=2:6:1] \
    "feeds.data" using 1:(sum [col=i:6] column(col)) \
      notitle \
      with lines lc rgb "#000000" lt -1 lw 1

This results in the following graph:

Gnuplot stacked graph with outlines between layers

Gnuplot stacked graph with outlines between layers

Note that you can change the lines’ attributes such as colour with the lc rgb colour command; the type with the lt type command (where -1 is a solid line); the width with lw width command.

You can determine the default line types for a particular terminal type by replacing the plot command with the test command. The Gnuplot manual states:

The default linetypes for a particular terminal can be previewed by issuing the test command after setting the terminal type. The pre-defined colors and dot/dash patterns are not guaranteed to be consistent for all terminal types, but all terminals use the special linetype -1 to mean a solid line in the primary foreground color (normally black).

Doing so with the png terminal gives the following:

Gnuplot build-in test graphic

Gnuplot build-in test graphic

The right-hand side of this test image gives the default line types.

Iterating With OpalORB

If you’re using the Perl CORBA interface, OpalORB, and you want to iterate through a list then you have to use iterator objects and process the list some elements at a time.

Here is a method I’ve used for iterating in Perl:

# list all XML files in this folder
my $numPerGrab = 25;
my ($objectItems, $objectItemIterator); # scalars to hold iteration objects
$folder->list_with_patterns(
    +[ "*.xml" ], # pattern_seq
    $numPerGrab, # number per grab
    \$objectItems,
    \$objectItemIterator
);

my $next_n_result = 1;
while ( 1 ) {
    # process this chunk of returned items
    foreach ( @{$objectItems} ) {
        printf( "  - file name \"%s\"\n", $_->name() );
    }

    last if ( ! $next_n_result ); # no more
    last if ( $objectItemIterator->_get_orb ); # iterator not valid

    # get next set of items
    $next_n_result = $objectItemIterator->next_n( $numPerGrab, \$objectItems );
}

# should probably destroy the iterator when done with it
# - but don't know if checking _repo_id is the right way to do it
if ( $objectItemIterator->{_repo_id} ) {
    $objectItemIterator->destroy();
}

The equivalent iteration done in Java would look like:

ObjectItemsHolder objectItems = new ObjectItemsHolder();
ObjectItemIteratorHolder iterator = new ObjectItemIteratorHolder();

int numPerGrab = 25;

folder.list_with_patterns(
    new String[] { "*.xml" }, /* pattern_seq - the sequence of the patterns (ORed) */
    (int)numPerGrab, /* how_many - how many objects will be returned (0 for all) */
    objectItems, /*  items - the sequence of the subitems requested */
    iterator /* iterator - an iterator to retrieve the remaining subitems */
);

ObjectItemIterator recordSet = iterator.value;
bool next_n_result = true;
while ( true ) {
    for ( ObjectItem item: objectItems.value ) {
        System.out.println( "  - " + item.info.name );
    }

    if ( ! next_n_result )
        break;

    if ( recordSet == null )
        break;

    next_n_result = recordSet.next_n( (int)numPerGrab, objectItems );
} while ( 1 );

It is worth noting that when iterating it may be faster to get smaller groups of items than larger. Initially I was trying to pull 500 filenames at a time but I found my script actually performed much faster pulling 25 filenames at a time. A little counter-intuitive but worth testing depending on your circumstances.

There are probably better patterns to use for iteration and by all means leave me a comment if you can demonstrate a better one. But this should get you going if you have no idea where to start.

How To Get Object’s IOR With OpalORB

If you’re using the Perl CORBA interface, OpalORB, and you want to know how to get a text string representing the IOR of an object you can do the following (where session is the CORBA object I want the IOR of):

my $sior = $session->_get_orb->object_to_string( $session );

This is possible because almost all objects compiled from IDLs in Perl will inherit from the CORBA::Object class which has the _get_orb() method – and this returns a CORBA::ORB class instance which has the object_to_string() method.

A way of doing this in Java is equivalently:

String ior = session.toString();