This entry isn’t to announce that I’ve managed to do that; it’s the complete opposite.

I’ve found that Zenfolio has come on a long way since I looked at it a year ago.  The Zenfolio service offers custom theming, order fulfillment, password protection, digital licensing and more.  I could, in time, write my own code to do this, but I’m coming to the realisation that I just don’t have the energy, nor skill, to write my dream system.  I could probably get my programming up to the skill level required to do it properly, but the time I’d spend doing that could be spent listening to live music or processing photos from a gig so that I can attempt to earn a little pocket money from them.

I think I’ll take the live music and pocket money.  I get plenty of chances to excercise my coding streak at work.

The majority of the arguments we need to configure take the form
-D$variable=$value
-XX:$variable=$value

The problem is, Java also has arguments that look like
-X$variable$value

This means my nice simple recipe to deal with tweaking the Glassfish JVM options doesn’t actually handle all the cases, so I now need to either go write a more complex one (and imbue it with knowledge as to what variables don’t have equal signs), or write a second recipe with a different name to handle these special snowflake options.

Feh.

Exception occured in J2EEC Phasejava.lang.IllegalArgumentException: Unknown ContainerTransaction type [Requires]
com.sun.enterprise.deployment.backend.IASDeploymentException: Error loading deployment descriptors for module [EJB FILE] — Unknown ContainerTransaction type [Requires]
at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:390)

Pinged one of the developers about that, and apparently it means that the transaction-type in ejb-jar.xml is wrong.  Yay.  For a reason I cannot fathom, Google had no results for this error either.

Lo and behold, the culprit was revealed – the search went something like

UPPER(eventlogmsg) LIKE ‘%value%’

Even if that field was indexed,the use of the ‘%’ on the front would negate the use of the index.  I threw the query into PGAdmin, and discovered that the query plan that PostgreSQL chose was a pair of nested loop joins – unpleasant to say the least.  A quick gander at the docs, and a few SQL statements later, and I had a full-text index on the eventlogmsg field.  Several test queries convinced me that it was much faster, so I threw the new query into PGAdmin and asked for the pretty query plan.  Two hash joins and a sort, and a query time of 31 milliseconds; or more than 5000 times faster.

So, I’ve filed a Bugzilla entry for this with the OpenNMS team – unfortunately it’s specific to PostgreSQL 8.3, but that’s something that can probably be determined at run-time and install-time.  Hopefully they agree that it’s a worthwhile performance change, as full-text indexes won’t actually help the LIKE usage – the code will have to change to generate new SQL statements.

Because we might have multiple domains set up inside of Glassfish, I opted for a setup similar to the Tomcat5 init script – check the basename of $0, and use that to determine which domain to boot up.  The fiddling in start() gets around the fact that Glassfish doesn’t seem to write a PID file out where we need one.

So, just in case anyone else needs to do this:

#!/bin/bash
# chkconfig: 2345 85 15
# description: GlassFish is a Java Application Server.
# processname: glassfish
# pidfile: /var/run/glassfish.pid
 
# source function library
. /etc/init.d/functions
 
RETVAL=0
GLASSFISH_BIN="/var/lib/glassfish/bin"
 
# Basename works with symbolic links.
NAME="$(basename $0)"
unset ISBOOT
# Trim off the Sxx/Kxx prefix
if [ "${NAME:0:1}" = "S" -o "${NAME:0:1}" = "K" ]; then
    NAME="${NAME:3}"
    ISBOOT="1"
fi
# Trim off the glassfish- prefix
NAME=${NAME:10}
 
# /etc/init.d/glassfish should never be called directly.
if [ -z $NAME ]; then
        echo -n $"Cannot start Glassfish without specifying a domain."
        failure
        echo
        exit 1
fi
 
start() {
        echo -n $"Starting Glassfish V2 domain $NAME: "
        daemon --user glassfish --pidfile /var/run/glassfish-$NAME.pid "$GLASSFISH_BIN/asadmin start-domain $NAME >/dev/null 2>&1"
        RETVAL=$?
        if [ $RETVAL -eq 0 ]; then
                PID=`ps U glassfish | grep $NAME | awk '{ print $1}'`
                echo $PID > /var/run/glassfish-$NAME.pid
                touch /var/lock/subsys/glassfish-$NAME
        fi
        echo
}
stop() {
        echo -n $"Shutting down Glassfish V2 domain $NAME: "
        $GLASSFISH_BIN/asadmin stop-domain $NAME >/dev/null 2>&1
        RETVAL=$?
        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/glassfish-$NAME && rm -f /var/run/glassfish-$NAME  && success || failure
        echo
}
 
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart|reload)
        stop
        start
        ;;
  condrestart)
        if [ -f /var/lock/subsys/glassfish-$NAME ]; then
            stop
            start
        fi
        ;;
  status)
        status glassfish-$NAME
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|condrestart|status}"
        exit 1
esac
 
exit $RETVAL

The alternative is to define a /etc/sysconfig/glassfish file, and insert a variable with the list of domains to boot, in sequence.  This is a little harder to manage automatically in Puppet, but might be a better solution if precise boot sequences are required (this method will boot in sequence based on the S numbers in the base script, and then the alphabetical ordering of the names).

The most annoying part so far is that yesterday we encountered the hang situation 4 times, so I enabled a strace against the binary, and channeled the output across the ‘net to my PC where I’ve got a rolling 40,000 line buffer.  24 hours later, at a constant 2 Mbit/s, and we still haven’t hung.

I call Heisenbug.

Bummer.

I doubt Kerberos is going to solve this either, as the caching is performed by nscd, and it’s nscd that doesn’t cache the hashes.  I suppose I could work on a custom PAM module that hooks in to pam_ldap, and on successful authentication, stores a new MD5 password for the account in /etc/passwd.  This way, a machine going off-line would have accurate local password hashes, and authentication would still work.

That sounds like way too much work though, and outside the scope of my job.

I’ve spent quite a few hours over the past week or so cleaning up the Value Editor mod for the EVE Killboard, along with patches to the core code, such as implementing the ability to store raw mails and fixing the portrait grabbing code.  I’ve also spent several hours providing support on the killboard forum for various other users of the software, and I have to say, I’m quite content now.  There’s something about providing code (for free) that people find useful.

Not sure what I’ll tackle next (other than the value editor, it needs a few more tweaks) – perhaps the mod that shows ship fits as a graphic.  One of the forum users has reported that it whacks their Apache installation load wise.

Take two tables.  Un-optimized indexes are present on both tables.  EXPLAIN says 45 rows will be retrieved from one table, 5 from the other.  The 45 will invoke where, the 5 invokes temporary and filesort.

Alter the index on  the table returning 5 rows (it’s a pivot table).

Now explain says 75, 4, with 75 invoking where, temporary and filesort and 4 invoking where.

Alter the index on the table returning 75 rows so that the index hit picks up the data that’s getting pivoted.

Now explain says 158, 4 with 158 invoking where, index, temporary, and filesort and 4 invoking where.

The weird part?  The last modification is the fastest, even with no_sql_cache invoked.  More rows to be worked with (arguably it’s still a small number), but it’s faster.

Enter a legacy table from a few years ago that just hit the pointer limit. The only solution I had in my arsenal is to rebuild the entire table with a larger pointer size. The last time I did this, it took 3 days. 3 days to rebuild the indexes for a 4 GB table. This time around, I tried a method that involves creating a new table definition without the indexes but with larger pointers, copying in all the old data, copying the old index file to the new index file, and executing a repair table. Unfortunately, while this method is meant to be faster, it only works when you can fit all of the keys into memory, otherwise it uses a slower rebuild method. 2 GB of RAM isn’t enough to hold all of the keys :(

So, chatting with Dad, he started to say something that tickled the grey cells. Namely that MySQL has a merge table format that can merge two (or more) identical MyISAM files, and support reads from both underlying files, while specifying where the writes should go. This means I can create a new underlying table with the new pointer size, then overlay the merge and specify the new table as the target for the writes. Total time to perform – 10 seconds, most of which is typing! Much, much better than 3 days.