Today can thus be classified as a good day, even if it did involve a bug that caused a Catalyst 5500 core switch to crash, taking out our entire network.

Cisco support wiki on said bug.

• June 21st
• June 29th
• September 8th
• September 9th
• September 18th
• September 19th
• October 17th
• October 27th
• October 29th

Pretty much every one of those tickets has been ‘My Xen instance is unreachable’.  At least two of them are from disk issues on the underlying server, several have been file-system write errors in the Xen instance.

On the other hand, another Xen instance I take care of lives on a US Xen hosting facility.  It’s been running since May, and I’ve not had to raise a single ticket about it.

So, goodbye CheapVPS, hello Slicehost (and here’s hoping the Rackspace buyout doesn’t affect them negatively).

There were only a few hiccups with the migration too - primarily with maildrop.  Forgot to install the courier-authlib-devel package before compiling maildrop, so it insisted on reading /etc/passwd instead of the database that everything else refers to for e-mail.

Exception occured in J2EEC Phasejava.lang.IllegalArgumentException: Unknown ContainerTransaction type [Requires]
com.sun.enterprise.deployment.backend.IASDeploymentException: Error loading deployment descriptors for module [EJB FILE] — Unknown ContainerTransaction type [Requires]
at com.sun.enterprise.deployment.backend.Deployer.loadDescriptors(Deployer.java:390)

Pinged one of the developers about that, and apparently it means that the transaction-type in ejb-jar.xml is wrong.  Yay.  For a reason I cannot fathom, Google had no results for this error either.

You have two sides, one out in the field and one in. Each man that’s in the side that’s in goes out, and when he’s out he comes in and the next man goes in until he’s out. When they are all out, the side that’s out comes in and the side that’s been in goes out and tries to get those coming in, out.  Sometimes you get men still in and not out.

When a man goes out to go in, the men who are out try to get him out, and when he is out he goes in and the next man in goes out and goes in. There are two men called umpires who stay out all the time and they decide when the men who are in are out. When both sides have been in and all the men have been out, and both sides have been out twice after all the men have been in, including those who are not out, that is the end of the game.

Describes the game perfectly, don’t you think?

I knew I was on call, but said that I’d like to show up, so Matt kindly guest-listed me in.  With my new phone on hand, I decided a few days ago that I could probably make it, so long as I checked my SMS every 15 minutes or so (and it turned out that 3G signal in Stratford is about non-existent).

Set off with plenty of time to spare, which turned out to be a good idea, as Google’s map was utterly wrong about where Cox’s Yard was.  While pondering the map, I happened to look up, and what do I see but “Cox’s Yard” staring back at me from a building cat-a-corner from where I was.  A quick circuit of the one-way system that is the entrance to Stratford, and I was parked across the river.  Meandered across the bridge, over to the Yard, and hung around waiting for the gig to start.

A really nice part about this performance was that the Yard had limited the seating to roughly 90 people, despite having a capacity of 220+ for the main stage. Small tables had been laid out, with a few chairs and a candle per table.  With the lights turned down, it made a very cosy, intimate atmosphere for listening to some fantastic music.  Katie opened the night, with a new song (America), and then continued with a few of her works that I’d heard before at the GW Festival (including Tongues, Bus Stop and Other Places).  My impression was that she’s still a bit nervous in front of a crowd (even a small one), but it didn’t detract from her performance at all.

After Katie finished, Matt introduced the Street Shepherd Band, who launched right into some wonderful music, including a piece that (paraphrased) ’should never be played outside of a gypsy wedding; seeing as I’ve been playing it for nearly 20 years, that probably explains a lot of things!’. One of the more unique pieces they played had hints of Miles Davis in it - certainly different, but in a good way.  For the second set, Ana Garcia provided the dance for the flamenco, and I was blown away by the energy she brought to the performance.  I am, unfortunately, no expert on flamenco, but I can certainly believe that she’s one of (if not the) top flamenco performers in England.

I now have a standing invitation from Matt to any of the events he’s organising, an invitation that I intend to take up as often as I can.  The photo work was fairly hard without a flash (I enquired ahead of time whether flash or flashless was preferred), and some of the lighting played havoc with the D80’s sensor, but all in all it was an excellent session.  Rolled home around 11 P.M., after pausing to get a long-exposure shot from the bridge out over the river; trying to capture the mist and the swans.

A Most Excellent evening.

I just need to get pocket PuTTY installed, and get the VPN connection to $work functioning, and I’ll have an SSH and RDP capable device that can connect from anywhere with a WiFi or 3G signal (in the UK at least, the Pro apparently isn’t quad-band).

Lo and behold, the culprit was revealed - the search went something like

UPPER(eventlogmsg) LIKE ‘%value%’

Even if that field was indexed,the use of the ‘%’ on the front would negate the use of the index.  I threw the query into PGAdmin, and discovered that the query plan that PostgreSQL chose was a pair of nested loop joins - unpleasant to say the least.  A quick gander at the docs, and a few SQL statements later, and I had a full-text index on the eventlogmsg field.  Several test queries convinced me that it was much faster, so I threw the new query into PGAdmin and asked for the pretty query plan.  Two hash joins and a sort, and a query time of 31 milliseconds; or more than 5000 times faster.

So, I’ve filed a Bugzilla entry for this with the OpenNMS team - unfortunately it’s specific to PostgreSQL 8.3, but that’s something that can probably be determined at run-time and install-time.  Hopefully they agree that it’s a worthwhile performance change, as full-text indexes won’t actually help the LIKE usage - the code will have to change to generate new SQL statements.

Caused by: javax.mail.SendFailedException: Invalid Addresses;
nested exception is:
class com.sun.mail.smtp.SMTPAddressFailedException: 550 5.1.1 User unknown

at com.sun.mail.smtp.SMTPTransport.rcptTo(SMTPTransport.java:1130)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:525)
at javax.mail.Transport.send0(Transport.java:151)
at javax.mail.Transport.send(Transport.java:80)
at alt.javax.mail.TransportImpl.send(TransportImpl.java:18)
at com.atlassian.mail.server.impl.SMTPMailServerImpl.send(SMTPMailServerImpl.java:168)
… 7 more
X connection to localhost:10.0 broken (explicit kill or server shutdown).

What a way to end a day that hasn’t been one of my better ones (more on that later, maybe).  Having got Tomcat back up and running, and slimmed down to just JIRA, I was hoping that the day was going to smooth out, at least for the last working hour.  Instead, I’ve been watching Tomcat just fall over at random, spitting out an error that the X connection went away.  There is nothing in the Tomcat configuration that requires X, so I can only guess that Java noticed that I had an environment variable set, and even though there was no X at the end of the DISPLAY, decided to use it anway.  My only hypothesis is that after some period of time, Java determined that the connection wasn’t there, and terminated Tomcat.

Wonderful.

unset DISPLAY && ./startup.sh

In the week running up to last weekend, I did some frantic processing of the photos from the previous weekend, and shot them off to BonusPrint for a run of 8″x12″ glossys.  This turned out to be problem number one that week - the credit card payment system screwed up, so I had to call them and get things fixed.  What I didn’t realise (because it wasn’t made clear at the time) was that they had to put a hold on the print job indicating that the prints had to go to their office first… I also ran off some business cards using Vista Print, even though I knew they wouldn’t arrive in time for the weekend (they actually arrived yesterday).

With Friday in sight, and no sign on the proofs, I started panicking slightly.  My options were either to go into town the next morning, and hope that the local Jessops had printers on site, or to make a hurried purchase of a home-level photo printer capable of A4 prints.  In the end, I decided to gamble on Jessops, which worked out ok - they had printers on site, and were able to run off some 9″x6″ lustres for me in 30 minutes, though the black and white photos weren’t quite black and white (additive colour processing).  Tossed them in my bag, and headed for home to find all my gear.

Problem number two was not having the D80, lens, batteries, flash etc prepped the night before (actually, they should have been done by Wednesday).  Problem number three, which didn’t bite me until the evening, was that I didn’t have a checklist, and transferred my gear from one bag to another.  More on that in a bit.

I finally set off for the gig around 12:30 or 13:00, on my bike.  Problem 4 bit me as I was coming up to the bridge that crosses the River Leam on the way - I changed the front and rear cogs too close together, under the strain of an incline.  There was an almighty racket, a clunk, and then the pedals stopped moving; as I stopped pedalling, there was the unpleasant sound of a chain falling off of the cogs.  Yep, I dropped my chain.  Off the bike, prop it up, fiddle with the chain, crank the pedal and I was back in business, albeit with dirtier fingers than before.

Oh yes, networking.  Well, 5 of the groups that performed the previous weekend were at the GW Fest, so I took the proof prints with me, in ‘nice’ manila envelopes, and gave them to the bands.  I’ve also told them that they’re free to use any of the photos from the weekend, under a non-exclusive license, and to contact me if they want higher resolution versions.

Whether anything will come of it, I don’t know, however I’m going to start stepping up a gear.  The business cards are done, and I’ll be bringing a .com on-line to sport a portfolio of my photos.  I already have a gallery for my ‘mundane’ photos, but nowhere to showcase the ones I think are really good.  The design is already sketched out, I just need to decide on how to host the actual images - do it myself, or use a site like SmugMug or ZenFolio.

Back to the music, I mentioned problem number three earlier.  As the night started to draw in, I was shifting my ISO ever upwards.  Around ISO 1600 is where I like to stop, otherwise the noise does start to get a bit distracting.  So, time to run the flash in TTL mode, except I couldn’t find the batteries for the flash.  Yep, in the rush to transfer my gear from bag to bag, I forgot to transfer the batteries.  More haste, less speed.  I also forgot to grab my ear plugs, which are kind of essential when a band like the Satin Dolls are closing the gig - they’re loud.  Loud, but very good.

I stayed for the indoor jam session featuring Katie Dove-Dixon (pictured) and Kristy Gallacher, and finally rolled home (on a working bike) sometime around midnight.  11 hours of music, and most of it spent on my feet.  Good time though, just 400 photos to filter and process!