Gotta love trying to work out why something is crashing when no one has said that the configuration has been changed (and the other implementations are perfectly stable).

Note to self: tripwire those machines.

So simple, and now I have full visibility of the fans and temperatures on the host server.

It turns out the proper way to do it, when using Exchange integration, is to access Tools > Accounts, and add the calendar subscription there.  The Send/Receive option in Outlook then works flawlessly, pulling down all the changes in the SugarCRM source every time Send/Receive is run.

The majority of the arguments we need to configure take the form
-D$variable=$value
-XX:$variable=$value

The problem is, Java also has arguments that look like
-X$variable$value

This means my nice simple recipe to deal with tweaking the Glassfish JVM options doesn’t actually handle all the cases, so I now need to either go write a more complex one (and imbue it with knowledge as to what variables don’t have equal signs), or write a second recipe with a different name to handle these special snowflake options.

Feh.

Today can thus be classified as a good day, even if it did involve a bug that caused a Catalyst 5500 core switch to crash, taking out our entire network.

Cisco support wiki on said bug.

Exception occured in J2EEC Phasejava.lang.IllegalArgumentException: Unknown ContainerTransaction type [Requires]
com.sun.enterprise.deployment.backend.IASDeploymentException: Error loading deployment descriptors for module [EJB FILE] — Unknown ContainerTransaction type [Requires]
at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:390)

Pinged one of the developers about that, and apparently it means that the transaction-type in ejb-jar.xml is wrong.  Yay.  For a reason I cannot fathom, Google had no results for this error either.

Lo and behold, the culprit was revealed – the search went something like

UPPER(eventlogmsg) LIKE ‘%value%’

Even if that field was indexed,the use of the ‘%’ on the front would negate the use of the index.  I threw the query into PGAdmin, and discovered that the query plan that PostgreSQL chose was a pair of nested loop joins – unpleasant to say the least.  A quick gander at the docs, and a few SQL statements later, and I had a full-text index on the eventlogmsg field.  Several test queries convinced me that it was much faster, so I threw the new query into PGAdmin and asked for the pretty query plan.  Two hash joins and a sort, and a query time of 31 milliseconds; or more than 5000 times faster.

So, I’ve filed a Bugzilla entry for this with the OpenNMS team – unfortunately it’s specific to PostgreSQL 8.3, but that’s something that can probably be determined at run-time and install-time.  Hopefully they agree that it’s a worthwhile performance change, as full-text indexes won’t actually help the LIKE usage – the code will have to change to generate new SQL statements.

Because we might have multiple domains set up inside of Glassfish, I opted for a setup similar to the Tomcat5 init script – check the basename of $0, and use that to determine which domain to boot up.  The fiddling in start() gets around the fact that Glassfish doesn’t seem to write a PID file out where we need one.

So, just in case anyone else needs to do this:

#!/bin/bash
# chkconfig: 2345 85 15
# description: GlassFish is a Java Application Server.
# processname: glassfish
# pidfile: /var/run/glassfish.pid
 
# source function library
. /etc/init.d/functions
 
RETVAL=0
GLASSFISH_BIN="/var/lib/glassfish/bin"
 
# Basename works with symbolic links.
NAME="$(basename $0)"
unset ISBOOT
# Trim off the Sxx/Kxx prefix
if [ "${NAME:0:1}" = "S" -o "${NAME:0:1}" = "K" ]; then
    NAME="${NAME:3}"
    ISBOOT="1"
fi
# Trim off the glassfish- prefix
NAME=${NAME:10}
 
# /etc/init.d/glassfish should never be called directly.
if [ -z $NAME ]; then
        echo -n $"Cannot start Glassfish without specifying a domain."
        failure
        echo
        exit 1
fi
 
start() {
        echo -n $"Starting Glassfish V2 domain $NAME: "
        daemon --user glassfish --pidfile /var/run/glassfish-$NAME.pid "$GLASSFISH_BIN/asadmin start-domain $NAME >/dev/null 2>&1"
        RETVAL=$?
        if [ $RETVAL -eq 0 ]; then
                PID=`ps U glassfish | grep $NAME | awk '{ print $1}'`
                echo $PID > /var/run/glassfish-$NAME.pid
                touch /var/lock/subsys/glassfish-$NAME
        fi
        echo
}
stop() {
        echo -n $"Shutting down Glassfish V2 domain $NAME: "
        $GLASSFISH_BIN/asadmin stop-domain $NAME >/dev/null 2>&1
        RETVAL=$?
        [ $RETVAL -eq 0 ] && rm -f /var/lock/subsys/glassfish-$NAME && rm -f /var/run/glassfish-$NAME  && success || failure
        echo
}
 
case "$1" in
  start)
        start
        ;;
  stop)
        stop
        ;;
  restart|reload)
        stop
        start
        ;;
  condrestart)
        if [ -f /var/lock/subsys/glassfish-$NAME ]; then
            stop
            start
        fi
        ;;
  status)
        status glassfish-$NAME
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|restart|condrestart|status}"
        exit 1
esac
 
exit $RETVAL

The alternative is to define a /etc/sysconfig/glassfish file, and insert a variable with the list of domains to boot, in sequence.  This is a little harder to manage automatically in Puppet, but might be a better solution if precise boot sequences are required (this method will boot in sequence based on the S numbers in the base script, and then the alphabetical ordering of the names).

This week I’m on call.  I had to VPN in to check out a failed Internet connection, and found to my horror that DNS resolution no longer worked.  I pondered for a moment, and then hard-coded the DNS server for the underlying network device to the work DNS servers and was able to get on with the job.

I’ve just been looking for what the heck has gone wrong; my home PC is able to open up the same VPN connection (I checked every single setting) and DNS resolution works.  What I found was a page describing a registry edit that fixes the problem.  Did said regedit, and presto, the VPN from the laptop works again.  The thing that bugs me is I don’t know how that setting got changed.  Mindboggling.

For the record, in case that page goes away, edit HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Linkage, double-click Bind and move the NdisWanIp entry to the top of the list.

Windows.  Feh.

Port  Status     Channel   Channel     Neighbor                  Neighbor
                 mode      status      device                    port
----- ---------- --------- ----------- ------------------------- ----------
8/7  connected   on        channel
8/8  connected   on        channel

it means the two ports are bonded together.  This poses distinct problems when the server isn’t aware that the two connections are bonded.  A CentOS installation process that takes 5 minutes normally was still running 3 hours later, and it wasn’t even half-way.  What was even more interesting was that the repository server showed files being downloaded, but Anaconda swore blind that nothing was happening.

Live and learn.