Update on email delays
October 25th, 2006 12:35 AET

Yesterday morning a second issue arose with the data repository that enables Bayesian analysis of incoming spam. Customers experienced delivery delays and may have received significantly more spam than usual as a consequence of the earlier failure.

Work is ongoing to introduce a second database to augment the existing database, eliminating this single point of failure within the spam infrastructure.

Update 27/10/2006 17:50 AEST: The second database server for Bayesian analysis has now been enabled on the Ilisys mail cluster.

Posted in Unscheduled | Comment »

Ilisys website/toolkit outage
October 20th, 2006 10:54 AET

The Ilisys website and toolkit were offline for a period between 4:34 AEST and 8:34 AEST. Note our own website and toolkit are on a standalone server, not a clustered environment. We’ve had some reports of a small number of customer websites with issues; it appears these are unrelated but we are still investigating.

Update 15:30 AEST: Sites displaying the Ilisys standard holding page would have appeared to be down as the stylesheet on that page was loaded from ilisys.com.au. We will address that issue.

Update 17:15 AEST: The Ilisys standard holding page stylesheet issue (above) has been resolved.

Posted in Unscheduled | Comment »

Mail delays
October 20th, 2006 10:30 AET

Our servers are currently processing a backlog of mail that has built up on our mail servers overnight. Technicians are working to reduce the mail queue as quickly as possible.

Update 11:45 AEST: Mail queues have reduced by one third and are now processing faster.

Update 13:28 AEST: Email received now will be delivered via an alternate mailserver within 15 minutes of receipt. Mail queues on the backlogged mailserver continue to decrease.

Update 15:15 AEST: Mail queues are back to normal.

Posted in Unscheduled | Comment »

Mail delivery delays
October 17th, 2006 15:07 AET

We have been experiencing some mail delivery delays on our mail servers today due to excessive amounts of spam. We are working to reduce the amount of email in our queues and will be bringing online new hardware to help cope with the extra load.

Posted in Unscheduled | Comment »

MySQL4 Update
October 10th, 2006 10:34 AET

The issue with mysql4.ilisys.com.au appears to have been resolved. The issue was a corrupt innodb table that caused a MySQL segmentation fault. We will be reporting this bug to MySQL.

Part of the troubleshooting process was to move mysql4.ilisys.com.au to other hardware and Linux distribution to isolate the issue. We will be migrating the server back to original hardware at 20:00 AEST which will require an outage of ~5 minutes in a 15 minute outage window.

Posted in Unscheduled | Comment »

mysql4 errors
October 9th, 2006 11:41 AET

The mysql4.ilisys.com.au server is having a problem and is being restarted now. Investigation into the problem is ongoing.

Update 12:25 AEST: We have managed to restart the server and rebuild the InnoDB databases and are monitoring the server closely to make sure the problem doesn’t re-occur.

Update 13:35 AEST: Although mostly up, the mysql4 server is crashing and restarting on a periodic basis. We are working to discover what the problem is.

Update 15:57 AEST: We have been unable to resolve the problems with the mysql4 server and are presently building a new server to replace it. Currently mysql4 is up but is restarting every few minutes causing timeouts. We expect the cutover to the new server to happen around 17:00 AEST. We apologise for this problem and are working as fast as possible to get it resolved.

Update 18:17 AEST: The cutover has occured to the new server now. However we are still experiencing issues with the server crashing and restarting on a periodic basis. We are currently investigating other options to fix the problem.

Update Tue 10:06 AEST: We have identified what we believe the problem on mysql4 to be. We are currently working to fix this problem and hope to have the system back to a stable condition shortly.

Posted in Unscheduled | Comment »

Some email delivery delayed
October 6th, 2006 11:22 AET

A large email queue has developed on one of our spam processing machines.  We are working to get this email delivered as soon as possible.

Posted in Unscheduled | Comment »

MySQL Maintenance
October 6th, 2006 00:58 AET

The server, mysql.ilisys.com.au (203.202.10.12/203.202.10.75), was down for maintenance between 0:13 and 0:19 AEST.

Posted in Scheduled | Comment »

Delivery delays on some email
October 4th, 2006 09:10 AET

One of our spam processing servers has a longer than normal mail queue resulting in delays on delivery of some email overnight. We are working to clear this queue as soon as possible.

Update: 11:36AEST – Our mailservers are processing (and rejecting) significant quantities of spam, which is causing other email to be delayed.  We are working to keep the queues as low as possible, and to bring the delivery times back to normal.

Posted in Unscheduled | Comment »