In light of the above requirements, we chose a solution based on a dedicated PC running UNIX® with a multiport serial card, and some software designed to handle serial consoles.
It includes the following elements:
A surplus PC. We used a Pentium® 166, with a PCI bus, 2Gbyte hard disk and 64Mb of RAM. This is a massive overkill for this task, and P-100, 500Mb, 32Mb would be more than enough.
A PC UNIX system. We used FreeBSD 4.3 as that is used for other tasks within our office.
A multi-port serial card. We chose the EasyIO™ PCI 8-port card from Stallion Technologies. This cost us about $AUD740, or under $100/port, from Harris Technologies (which has lots of stuff but is by no means the cheapest place in town - shop around and you might get it a lot cheaper). This card has a big DB80 connector on the back, and a cable plugs into that which has a block with 8 RJ-45 sockets on it. (We chose the RJ-45 version as our entire cable plant is RJ-45. This allows us to patch connections from the required box to the console server without any special cables.) This is the only thing we needed to buy to make this all happen.
We build two servers, one for each computer room, with 8 ports in one and 16 ports (via two EasyIO PCI cards) in the other. If we needed more than 16 ports, then another of the Stallion cards would be more cost-effective. We could conceivably support 128 ports in each server (with 2 EasyConnect 8/64 host cards and 8 16 port RJ-45 modules) for about $AUD12,000.
A modem for remote access to the console server host when the network is down. We have not done this yet as the computer room is next door, but when we put a server in Sydney we will add the modem. The idea is that when the network is down, you can dial up and log into the server machine and run the console program locally. For security, we will probably leave the modem powered off and ask the gopher in Sydney to turn on the well-labelled button when we need it.
A program called conserver. This program does all the magic required to enable remote access to consoles, and do the replaying and logging etc. It comes in two parts: a server called conserver that runs as a daemon and connects to the serial ports, handles logging etc, and a client program called console that can connect to the server, display console messages, send keystrokes (and BREAK), etc.
This design covers all the major requirements except remote power cycling:
Remote access comes because the console client program works across the network.
Logging is handled by the conserver program.
If the network is down, then we can use the console on the PC to run the console client locally. For remote sites, we can add a modem for dial-in access to the server command line to run the client.
By patching the Solaris™ servers (see Section 6), we can avoid pranging the whole computer room when the console server PC crashes (or the power supply fails, or whatever).
We already have pager alerts from another system we have installed, but the console server has all the required log info so that could easily be implemented if we needed. And it even has a modem for calling the pager company!
We do not currently support remote power cycling. Some versions of the conserver program support this, but it does require specialised serial-controlled power boards. We have no immediate need for remote power cycling (we have a gopher in each remote office who can do it by remote control) so this is not a major problem, and we could add it easily should we ever see the need and get the appropriate hardware.
This solution was very cheap. Total cost for the 9-port server was $AUD750 for the IO card, as we re-used a surplus PC and already owned the hardware for the special cables. If we had to buy everything, then it would still only cost around $AUD1500 for the 8-port server.