Director’s Report: Easter 2015

Dear Members,

My name is Daniel Watts and I am the Managing Director of Aluminati. Every once in a while I write a report to address our members; covering pertinent topics of the recent past as well as those critical to our future.

In this report:

A Rocky Month – Report on Recent Service Interruptions
New Infrastructure and Performance Enhancements
Problems with Spam Blacklists
Abuse of Trash Quotas
Happy Easter

A Rocky Month – Report on Recent Service Interruptions

For a period of around 3 weeks from the middle of February to the middle of March our service experienced a period of notable instability with issues occurring every 1-3 days. Faults ranged from general system outages, SSL certificate errors, missing password resets, delayed billing approvals and email client connection problems.

First of all, I would like to thank you for your patience during that troubled time. Being unable to fully utilize one’s email account is undoubtedly inconvenient as email forms such an important part of our daily life. I am sorry for any frustration caused to those of you caught up in those problems.

The reason for the disruptions are to do with some major systems that finally took place in February. This was the first phase of a large scale server infrastructure upgrade. The last time we did such a major upgrade was in Summer 2009 and it is time again to invest in new hardware to support the growth and future of Aluminati’s services.

Despite many months of planning and testing, there were a variety of problems that did not present themselves until stressed in the ‘real-world’ environment. The majority of these were to do with our having to run a split-database system with database clusters having to talk to each other across datacentres. We needed to optimize various parts of the codebase to account for the longer query times. Other problems were due to having to issue new SSL certificates and these not functioning correctly in certain browsers (especially Firefox) despite conforming to standards. Finally the tightening up of our security caused many of you to stop receiving emails to your email programs – whilst we have always published our IMAP/POP servers as starting with “mail.” “imap.”, many programs default to “imap.”, which used to work. Once we tightened security, these unsupported addresses were blocked.

All in all, over the three weeks following the upgrade work we uncovered 2-3 dozen issues that were all corrected. We are now in a much healthier state as a result and, whilst we will still have the occasional interruption to service, such events are now down to a much lower frequency with recovery times in the range of 10-20 minutes.

New Infrastructure and Performance Enhancements

I am sure you will wonder if this disruption is worth it. I would like to assure you it will be. Just like our current infrastructure was a major step-change from pre-2009, this move will allow us to bring to bear the latest techniques in service performance and stability. We are investing tens of thousands of pounds in new equipment and taking the opportunity to implement more modern storage formats. We are also intending to implement instant failover which would reduce our 10-20 minute recovery times to perhaps 10-20 seconds (or less), effectively eliminating 90%+ of all noticeable disruptions.

We are also going to be implementing a second methodology which will allow our regular daily-active users to benefit from hugely faster hardware. Many of our members use our email service as a long term archive of their student emails – which is absolutely fine. However, these inactive accounts take up a significant amount of storage that does not require fast access. We will be implementing an intelligent tiering system so that inactive accounts can be moved to long-term secondary storage, whilst active accounts will be on our fastest SSD disks. Our active members will experience a faster and more responsive service as a result.

Problems with Spam Blacklists

A growing problem over the last year has been the increasing tendency of large organisations to outsource the handling of their spam blacklisting to 3^rd party blacklist management services and for those outsourcing companies to become decreasingly responsive. Unfortunately with over 120,000 email accounts, we do occasionally have users who are compromised by spammers – be it a weak password or a virus on a PC. Whilst we can generally freeze any account exhibiting spammy behaviour, this can sometimes cause blacklisting companies to list our IP addresses. Indeed some of them will block an entire range of IPs even if the actual offending server is nothing to do with us.

Abuse of Trash Quotas

To date we have been quite generous with the amount of email you can keep in your email trash folders. Originally this was so that even if you were over quota, you could always have space in Trash to move email to. We expected our members to regularly ‘Purge’ their trash, thus permanently deleting their old email. We have found a very small number of users, perhaps unknowingly, have not purged their trash and as a result have built up many gigabytes of email. Our published policy of removing Trash emails older than 180 days has never actually been enforced but will soon be coming in to play. This should not affect any of you using Trash appropriately. However any member deliberately abusing their Trash as an unlimited storage location should take note and move any valuable email into ‘normal’ folders for long term storage. You can of course add extra storage to your account at any time.

Happy Easter

Finally, I hope you have all had a very Happy Easter. As a small ‘Easter egg’ in recognition of the recent disturbances to your email service and to your patient and ongoing support, I have asked my team to credit a month of service to all of you, moving your renewal days forward by 30 days next year. This 30 day credit will be applied over the next 14 days. I continue to feel very lucky to be in a position to serve a unique membership of top graduates and I look forward to completing these service upgrades to further enhance the service we deliver to you.

Warm regards,

sig2