This post should be taken with a huge fist of salt, it’s written after the loss of many hours over the weekend
Doing system related work exposes you to a whole lot of mistakes & stupidity, you have to have a tough skin & in some situations being able to cope with the fickle nature of the relationships maintained will help too.
Personal relationships aside, tools, technology and vendors are far more infuriating in my opinion, If one attains interpersonal skills they can deal with maintaining relationships, vendor requiring an active support contract for a closed source “enterprise” tool? *shrug*
I am dramatizing ofcourse, lets take something like a firmware upgrade, ok, but pretend you’re not a HP or Oracle customer too in that case, right, so you have a system which needs a firmware update applied to it & you’re not running MS DOS, Windows or Linux, be prepared for a lot of wasted time. Infact, never mind the updates, just using the tools is a time sink. Being at the mercy of very long feed back loops & small windows for events to happen is infuriating and things don’t seem to be getting much better from the vendors side and as things move forward the systems on older software become increasingly more difficult to manage.
Dell & Supermicro recently are a particular pain of mine as they’re my general choice for server hardware but I’ve also experienced just as much pain with Sun & HP hardware.
Between the two former hardware vendors I’ve lost countless amount of time, reasons from the requirement of MS-DOS or Windows to write floppy disks to functionality only operational in Windows and many others things that I’ve managed to mask away the pain of at this present moment so that I can’t recall. As things are not getting any better the amount of time lost accumulates, for example, lets take a chassis loaded with 24 disks, a SAS controller & 32GB of RAM. To POST such hardware takes between 12 to 15 minutes, upon intialistation you have the opportunity to make a selection of entering the BIOS or selecting an alternative boot method temporarily for this attempt, if you’re using the remote console functionality there’s a big delay between what’s shown on the remote console window & what’s actually happening on the system, the difference being large enough to miss the opportunity to make a selection and unfortunately you wont know if your selection was successful until the end of the POST, at which point you have no choice but to restart & try again. If you wish to enter the BIOS to change a setting, you have to endure POST before entering the BIOS. After 4 subsequent reboots during POST the system will reset the BIOS back to factory default settings which will add a few more minutes as everything re-initializes from scratch. In one particular scenario the timeout for the OS boot loader had also been switched off on system, I was trying to debug an OS related issue which needed me to interrupt the boot loader to set some options, then boot the kernel.
Workflow went as follows:
System panics > reset via IPMI
Wait for Post & as it comes to end, press space franticly to interrupt boot
If too late, reboot & try again
If by 4th attempt still haven’t managed to boot system as intended BIOS is reset to factory default settings so on this iteration you don’t attempt boot but re-enter the BIOS to put the settings back to how you require them to be.
I wonder what the total amount of time wasted during POST/boot on the X8 series worldwide is?
Or how about with Dell or Sun, attempting to boot a server remotely using the virtual media functionality (written in Java) does not work on Mac OS X but does on Windows, doesn’t work on IE but does on Firefox after the security levels in Java have been reduced. For Dell, this is a big step forward, previously it was all ActiveX.
Intel still ships the tools to manipulate the firmware settings of network cards as DOS binaries, leaving the user to generate the media to run them off. I’d go as far as to say integrate them onto a boot disk to start the system with, In 2014, DOS is not commonly found in peoples tool belts.
How does one turn this around? The coreboot project is promising, vendors like PCengines have adopted it for their new hardware platform but still far from mainstream adoption, I’ve only dealt with IBM server hardware once in the past at a customers, the system was self sufficient, all firmware updates came on bootable media already prepared, just needed to write the image & boot the system from the media.
Perhaps there could be a new initiative in the IT industry to change from “nobody ever got fired for buying from $VENDOR” to “nobody ever got hired ever again for writing $X for $VENDOR” on the grounds of “crimes against operations” by wasting peoples lives through poorly implemented, clumsy crap which is then utilized in a critical area.
Hardcoded assumptions are never acceptable!!!