Scheduled network outage
April 29th, 2008 15:10 AET

Impact: A 1 hour network outage
Services: All web and email services
Date: Wednesday 30 April 2008
Duration: 00:05 – 01:05 AEDT
Status: Resolved

We require a one hour outage window to build redundancy into our cluster load balancers. This is one of the remedial steps outlined in our report into the unscheduled outage on 31 March.

Update: 12:05 AEST – commencement of the scheduled outage is occurring as planned.

Update: 12:17 AEST – the upgrade is proceeding as planned, whilst there may be some short connectivity losses, there should not be any extended outages at this stage

Update: 01:15 AEST – normal service has been restored and the planned upgrade has been completed. Some final tests are underway which may cause a further two minute outage for some services. During the bulk of this work tonight most services would have only experience brief outages. The exception to this is the Power cluster, which experienced 15 minutes of intermittent connectivity. The cause of that has been successfully identified and has now been fixed.

Update: 01:39 AEST – final testing has been completed and all services are now operational.

Posted in Scheduled | Comment »

Sessions error on power cluster
April 23rd, 2008 16:35 AET

Impact: Session error on power cluster sites
Services: Power cluster
Date: 23 April, 2007
Duration: 14:55 AEST – 15:16 AEST
Status: Resolved

Sites using PHP sessions on the power cluster would have shown a session_start error message when loading certain web pages. This was due to a mis-configuration of a PHP setting. The issue was resolved shortly after it was noticed.

Posted in Unscheduled | Comment »

Email Delays
April 22nd, 2008 14:07 AET

Impact: Delay on incoming email
Services: Email
Date: 22 April, 2007
Duration: 12:22 AEST – 14:49 AEST
Status: Resolved

We are currently experiencing some issue with our incoming server that is causing a delay with email coming into our systems. The technicians are working on this fault and hope to have it resolved as soon as possible.

Update 14:04 AEST: Technicians have identified the problem causing the delay with email. We are currently updating our mail systems configuration to fix the problem.

Update 14:49 AEST: The issue effecting our incoming mail servers has now been resolved. Some mail would have been deferred back to the sending mail server however this mail will eventually be delivered. Ilisys apologises for the inconvenience.

Posted in Unscheduled | Comment »

Email delay
April 15th, 2008 10:31 AET

Impact: delay on incoming email
Services: email
Date: 15 April, 2007
Duration: 04:45 AEDT – 10:42 AEDT
Status: Resolved

One of the servers in our SMTP mail cluster has built up a large queue. Technicians are currently working on clearing the queue at the moment however there will be a delay on mail waiting on that server’s queue. Any new mail will be processed immediately.

Update 09:55 AEDT: The size of the mail queue on the affected server is coming down now. It has been taken out of our cluster so that it does not accept any new mail. The mail queue is expected to be back to normal within the next 30 minutes.

Update 10:42 AEDT: The email queue is back to normal now.

Posted in Unscheduled | Comment »

Issue with toolkit2.ilisys.com.au
April 3rd, 2008 18:22 AET

Impact: toolkit2.ilisys.com.au is inaccessible
Services: toolkit2.ilisys.com.au
Date: 3 April, 2007
Duration: 17:54 AEDT – 18:37 AEDT
Status: Resolved

There is currently an issue connecting to the toolkit2.ilisys.com.au website. Technicians are working on the issue.

Update 18:37 AEDT: The issue connecting to the toolkit2.ilisys.com.au website has been resolved.

Posted in Unscheduled | Comment »