Cricalix.Net

Had a report recently that our QA application servers kept falling over (Glassfish that is, not the hardware).    Started looking into it today, couldn’t find a sign of anything wrong.  Then one of the lead developers added a comment about how they’d seen this behaviour when the default Glassfish debug port (9009) gets pinged by our monitoring system.  Then the developer who reported the problem commented ‘oh yeah, we turned that on on Sunday’.

Gotta love trying to work out why something is crashing when no one has said that the configuration has been changed (and the other implementations are perfectly stable).

Note to self: tripwire those machines.

Recently upgraded one of our HP servers to VMware vSphere 4.0, and found that we couldn’t see the hardware status from the vCenter client.  All it would print was “Hardware monitoring service not responding, the host is not powered on”, despite the host certainly being powered on.  A VMware Communities posting suggested that it was either firewall related (nope), or that removing the host and re-adding it might work.  Turns out there was a third option - just disconnect and reconnect the ESX host.

So simple, and now I have full visibility of the fans and temperatures on the host server.

Someone at work has requested a way to access our SugarCRM system from their Outlook 2007 / PDA client, so that the meetings can be synchronized properly.  A bit of research led me to the Sugar iCal patch on SugarForge, and the Internet Calendars option in Outlook 2007.  Unfortunately, the first set of instructions I found said to use the Files > Data Files Management option to access the Internet Calendars.  This method didn’t work very well (actually, it didn’t work at all), as Outlook would never synchronize the iCal file (it always said pending), and would promptly lose the setting when I closed Outlook.

It turns out the proper way to do it, when using Exchange integration, is to access Tools > Accounts, and add the calendar subscription there.  The Send/Receive option in Outlook then works flawlessly, pulling down all the changes in the SugarCRM source every time Send/Receive is run.

I’m doing some final tuning work on the Puppet recipes for our Glassfish installation, and Java has reared one of its ugly heads again. In this case, it’s the whole management of the command line arguments for the JVM.

The majority of the arguments we need to configure take the form
-D$variable=$value
-XX:$variable=$value

The problem is, Java also has arguments that look like
-X$variable$value

This means my nice simple recipe to deal with tweaking the Glassfish JVM options doesn’t actually handle all the cases, so I now need to either go write a more complex one (and imbue it with knowledge as to what variables don’t have equal signs), or write a second recipe with a different name to handle these special snowflake options.

Feh.

On a good day, I learn something new.

Today can thus be classified as a good day, even if it did involve a bug that caused a Catalyst 5500 core switch to crash, taking out our entire network.

Cisco support wiki on said bug.

Doing more testing at work today, and decided to pickup the latest compiled output from the build server.

Exception occured in J2EEC Phasejava.lang.IllegalArgumentException: Unknown ContainerTransaction type [Requires]
com.sun.enterprise.deployment.backend.IASDeploymentException: Error loading deployment descriptors for module [EJB FILE] — Unknown ContainerTransaction type [Requires]
at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:390)

Pinged one of the developers about that, and apparently it means that the transaction-type in ejb-jar.xml is wrong.  Yay.  For a reason I cannot fathom, Google had no results for this error either.

$work uses OpenNMS to monitor our various devices (servers, switches, routers, printers etc), mostly via SNMP.  Today, while looking at the various events that had been recorded, I noticed that a relatively simple search was taking more than 2 minutes to process through ~250,000 event rows (plus associated rows in other tables).  I turned on query logging (log_statements = ‘all’ && service postgresql reload), and re-ran the search from the web interface.

Lo and behold, the culprit was revealed - the search went something like

UPPER(eventlogmsg) LIKE ‘%value%’

Even if that field was indexed,the use of the ‘%’ on the front would negate the use of the index.  I threw the query into PGAdmin, and discovered that the query plan that PostgreSQL chose was a pair of nested loop joins - unpleasant to say the least.  A quick gander at the docs, and a few SQL statements later, and I had a full-text index on the eventlogmsg field.  Several test queries convinced me that it was much faster, so I threw the new query into PGAdmin and asked for the pretty query plan.  Two hash joins and a sort, and a query time of 31 milliseconds; or more than 5000 times faster.

So, I’ve filed a Bugzilla entry for this with the OpenNMS team - unfortunately it’s specific to PostgreSQL 8.3, but that’s something that can probably be determined at run-time and install-time.  Hopefully they agree that it’s a worthwhile performance change, as full-text indexes won’t actually help the LIKE usage - the code will have to change to generate new SQL statements.