At the beginning of 2003 we had a CriticalPath mail system running on Solaris x86 plus a Redhat box for SMTP, Radius and DNS. The DNS and Radius services were constantly down and we were struggling with huge mail queues. There was an attempt to install CriticalPath for Linux into Redhat on an Intel box with a Megaraid card, but the disk latency was enormous and the mail application never really worked.
The first step depicted towards the "FreeBSD solution" consisted in migrating this hardware and commercial software to FreeBSD 4.8 with Linux emulation.
The FreeBSD operating system is well-known for its great stability, plus its pragmatism and common sense to put applications on-line thanks to its excellent Ports System. We consider its release engineering process to be easily understandable, while the users' community at the official mailing lists keeps a polite and civilized style when it comes to asking for support or reading other people's problems and solutions.
Another important feature is quick deployment. Fortunately, we could state our OS install policy around FreeBSD's great out-of-the-box capability. In a small company you sometimes need to run to a Datacenter and quickly setup a server for some service. In the last two years, Argentina.Com acquired around forty servers, most of them Pentium IV but also several double-Xeons and a few double-Opterons to be co-located in the Datacenters where we have dialup and hosting operation contracts. All of them run FreeBSD, ranging from 4.8 (there are a couple with two years uptime and zero trouble) til currently 6.0-BETA2.
The general policy for the operating system is to try to bring all servers periodically to the stable code branch by using RELENG_4, RELENG_5 and now RELENG_6. This regularity lets us be more prepared regarding possible exploits at the operating system or base software level, especially in web servers.
The first re-engineering step was to put in place two FreeBSD 4.8 boxes whose unique task was to be authoritative DNS for all our domains. The chosen software was Bind9. Those boxes were co-located in different datacenters, taking care that there was good latency between them to avoid zone transfer problems, and making it possible to deal with TTLs between 60 and 600 seconds to have quicker response in case of trouble.
Second step was to deploy two more boxes of the same class, again in different Datacenters, to only deal with Radius and recursive DNS. The Network Access Servers at the Telcos were configured to send Radius Authorization and Accounting to those servers, and to assign these recursive DNSs to dialup users.
The third “golden rule” never to put SMTP incoming and outgoing in the same servers. We deployed separate FreeBSD boxes with postfix for incoming and outgoing mail.
The email migration required careful planning due to the fact that we were going to migrate both mail front and back-ends. We first built a perimetral antispam and antivirus system in FreeBSD 4.x and 5.x based on postfix, amavisd-new, clamav and SpamAssassin. These systems were to deliver mails to both the old and the new system until the new back-end was in place. In the meantime, we added small FreeBSD NFS boxes to increase CriticalPath's mailspool, without any problem.
At the frontline of incoming mail, we put in place several MXs of the Argentina.com domain to filter dictionary attacks (attempts to forward mail to nonexistent users) as well as a black-list derived from SURBL that resulted in almost no false positives. The mails are then multiplexed to a cluster of double-Xeons and double-Opterons where we run amavisd-new with MySQL based white and black-listing. We discarded the use of Bayes and Autowhitelisting at the global level because of great quantities of false positives and false negatives. We instead defined a few spam levels going from the least to the most tolerant, each one with cutoff or discard levels. Every email with a score below the one associated with the selected spam tolerance goes to the user's Inbox. Emails between this level and the cutoff level go to a user's folder named Spam, and those above the cutoff level get discarded because it is a very obvious spam. For the sake of simplicity, we transparently associated the use of the Address Book with the antispam system, so that every personal contact gets automatically whitelisted.
With the introduction of Spamassassin 3.x, the DNS traffic to query global blacklists grew considerably, so we signed agreements with SpamCop, Spamhaus and SURBL to install public mirrors of their databases in our FreeBSD equipment. Thanks to these mirrors that cost us between 1 and 2Mbps in traffic, we were able to dramatically cut down Spamassassin latency.
At the 3rd level there is the delivery to the maildrops. As soon as we started building a new Cyrus-Imap back-end with MySQL authentication, we needed to multiplex incoming mail to users in both old and new maildrop formats. Finally, we managed to migrate hundreds of thousands of mailspools to the new Cyrus architecture using a great tool named imapsync, which is directly installable from ports. We also put perdition, a POP3 and IMAP proxy, in the middle to assure a transparent migration and distribution of mailboxes across several servers. Briefly, all information of where a user's maildrop is located resides in MySQL, and is being used by all software pieces in the chain.
With regard to the hardware for disk space, we currently use seven Cyrus-Imap loaded FreeBSD boxes with diverse hardware. The biggest are Pentium IV with 4G of RAM and 3ware cards in chassis with 12 hotswappable bays, organized in 3 RAID-5 units of 1 Terabyte each. The 3ware software sends you en email whenever the RAID is degraded -mostly because of a failing disk- and lets you rebuild the RAID with everything up and running. We use smartmontools in the cases where we have less redundancy, to have immediate alerts of disks with temperature problems or failing selftests.
As webmail software, we chose a commercial product named Atmail, which is available with perl sources and utilizes mod_perl. Under FreeBSD it is extremely easy to deal with perl modules, you do not even need to use the CPAN shell, you just have to choose the right port and run "make install". After several months of integration work, we integrated the Client-only version of Atmail that talks IMAP with our back-ends. We had to modify some parts of the code to adapt the product to our massive free environment, and to our antispam and antivirus perimeter, in addition to our specific customizations and translations.
With the adoption of FreeBSD, there was almost no additional effort necessary to setup a working Apache, PHP and MySQL environment in minutes. Even the upgrades from PHP4 to PHP5 were painless. The ports system was again extremely useful in these cases, and permitted us to do things like compress text and html contents in Apache with just a few lines of documentation. In addition, we have experienced excellent performance and rock-solid stability and uptime.