Ilisys Support Hours during Xmas/NY
December 21st, 2009 18:32 AET

Weekdays 8:00am – 8:00pm (EST)
Weekends 9:00am – 5.00pm (EST)

Christmas Eve 8:00am – 6:00pm (EST)
Closed Public Holidays & Sat 26 December

Posted in Scheduled | Comment »

DNS issue for mail.ilisys.com.au
December 11th, 2009 16:23 AET

Impact: mail.ilisys.com.au
Services: Incoming mail using mail.ilisys.com.au
Date: Friday, 11 December 2009
Duration: 16:04 – 16:26 AEST
Status: Resolved

Summary:
Technicians are investigating an issue related to mail.ilisys.com.au.

Summary
This issue has now been resolved.

Posted in Scheduled | Comment »

MySQL Outage
December 9th, 2009 11:43 AET

Impact: Scheduled MySQL maintenance
Services: Ilisys.com.au and dynamic websites using m4-mysql1-1
Date: Wednesday, 9 December 2009
Duration: 12:15-12:25, 13:45 – 14:00, 17:00 – 17:15 and 18:25 – 18:40
Status: Complete

Summary:
We require a five minute outage in a ten minute window to perform maintenance to this MySQL database server.
During the outage, dynamic websites will temporarily lose access to their databases and page loads may time out.

Update, 12:20: This maintenance is now complete.

Update, 13:30: A further 15 minute maintenance window has been scheduled at 13:45, for further works towards restoring the ongoing issue.

Update, 13:45: This maintenance is now under way, we will update again once complete.

Update, 13:53: This maintenance is now complete.

Update, 16:55: A further 15 minute maintenance window has been scheduled for 17:00, for further works towards restoring the ongoing issue.

Update, 17:00: This maintenance is now under way, we will update again once complete.

Update, 17:13: This maintenance is now complete.

Update, 18:20: A further 15 minute maintenance window has been scheduled for 18:25, for further works towards restoring the ongoing issue.

Update, 18:25: This maintenance is now under way, we will update again once complete.

Update, 18:40: This maintenance is now complete.

Posted in Scheduled | Comment »

Email delays
December 8th, 2009 16:33 AET

Impact: Delay on some incoming email
Services: Mail
Date: Tuesday, 8 December 2009
Duration: 14:00 AEDST – Current
Status: Resolved

Summary:
Following our earlier outage, we have observed large mail queues building up on our servers. Whilst we’re processing them as per normal, there is a significant volume to process and email messages may by delayed until later this afternoon.

Posted in Unscheduled | Comment »

MySQL & ilisys.com.au issue
December 8th, 2009 14:15 AET

Impact: MySQL errors on some hosted websites and the Ilisys website unavailable
Services: Ilisys.com.au and some customers’ dynamic websites
Date: Tuesday, 8 December 2009
Duration: 14:00 AEDST – 18:40 AEDST
Status: Complete

Summary:
Following our earlier outage, we are still experiencing some ongoing issues with our InnoDB storage engine that are affecting a small number of services, including www.ilisys.com.au. To resolve the issues we need to perform some further maintenance that may affect clients using the same sever. We apologise for any inconvenience caused by this disruption.

Update, 16:20: Technicians are still working on this issue at the highest priority. At this stage, only clients on the “m4-mysql1-1″ server are outstanding.

Update, 19:00: The www.ilisys.com.au website issue has now been resolved. There is an ongoing issue with a small number of mysql databases with innodb tables.

Update, 9/12/2009 8:30: Engineers continued to work through the night on this issue and have been unable to restore a small number of clients’ databases. We’ll send a detailed explanation to affected clients this morning.

Update, 11:20: Technicians are continuing to implement a range of remediation measures. At this stage we are still unable to confirm an ETA but will continue to provide updates throughout the afternoon.

Update, 15:00: Recovery measures have not yet been successful. Technicians are continuing work to restore affected clients’ sites.

Update, 16:00: Technicians are still working through alternative recovery approaches. Unfortunately we are not able to provide an ETA at this time.

Update, 17:15: Further works have been completed and we’ve observed some positive results. Technicians are testing to confirm that affected clients’ sites are operational.

Update, 17:30: The issues relating to MySQL innodb tables has now been resolved.

Update, 17:45: Testing has identified areas that still require attention. Work continues.

Update, 18:50: Further maintenance and testing has occurred. The issues relating to MySQL innodb tables has now been resolved.

Update, 10/12/2009, 07:35: Overnight we resolved our InnoDB database issue and affected websites are now functioning normally.

We will compile a formal fault report over the next 48 hours. In the mean time, we’d like to thank you for your patience as we’ve worked through this issue.

Posted in Scheduled | Comment »

Network connectivity issues
December 8th, 2009 10:06 AET

Impact: No connectivity to all services
Services: All
Date: Tuesday, 8 December 2009
Duration: 11:00 – 13:40 AEDST
Status: Resolved

Summary:
We are currently experiencing network connectivity issues for all services. Technicians are looking into the issue.

Update, 11:18: We have established that this issue is related to an outage at our data centre provider. We will update again once we have further information.

Update, 11:35: Thanks for your patience. Services are now coming back online progressively.

Update, 11:55: Engineers are bringing services back online carefully and in sequence. Currently, we estimate that all services will be restored within 30 minutes.

Update, 12:30: Some services are back online, including POP/IMAP mail. Technicians are still working to restore other services. We will provide an update within the next 15 minutes.

Update, 12:45: Technicians are still working to restore full service. A number of web services are back online, and we’re working to bring the rest back up. We will provide another update in the next 15 minutes.

Update, 13:05: Most services have been restored to partial or full capacity, including the Power cluster, Toolkit, Webmail, POP/IMAP and inbound email. Technicians are still working on restoring full capacity to services. We will provide another update in the next 15 minutes.

Update, 13:20: Further services have been restored, including the Standard cluster web sites. Some services are still operating at partial capacity, and we’re working to restore these to full capacity shortly.

Update, 13:40: All services have been restored to full capacity. Some email sent between 11:00 – 13:00 has been queued and may be delayed for a few hours.

Update, 14:05: We’ve observed some issues with some MySQL databases using InnoDB tables. Our engineers are working to fix this now.

Posted in Unscheduled | Comment »

Toolkit Outage
December 7th, 2009 10:49 AET

Impact: 10 minute outage inside a 15 minute window on Ilisys Toolkit and Website sales process
Services: Toolkit and Ilisys website sales process
Date: Monday, 7 December 2009
Duration: 16:15 – 16:30 AEDST
Status: Complete

Summary:
Following this morning’s outage, we require a brief outage window to perform maintenance on the servers housing our internal database.
During this time, the Toolkit and Ilisys sales process will be unavailable. We apologise for any inconvenience caused by this interruption.

Update, 16:20: The maintenance is now underway.

Update, 16:43: This maintenance is now complete.

Posted in Scheduled | Comment »

Toolkit & Standard Cluster issue
December 7th, 2009 08:06 AET

Impact: Sites hosted on Standard Cluster, access to Toolkit, sales and renewals, and Webmail
Services: Standard Cluster web sites, Webmail, Toolkit, and some Ilisys SSL pages
Date: Monday, 7 December 2009
Duration: 09:00 – 09:30 AEDST
Status: Resolved

Summary:
We have observed some intermittent issues connecting to Ilisys pages such as Toolkit, and our sales and renewals system. Technicians are currently investigating.

Update, 09:15: This also appears to be affecting some sites hosted on our Standard Cluster, as well as access to webmail.

Update, 09:30: Technicians are still investigating the underlying issue, although these services have been restored. We apologise for any inconveniences caused by this disruption.

Posted in Unscheduled | Comment »

System maintenance activities
December 4th, 2009 14:25 AET

Impact: Several Small outages as we perform maintenance
Services: Toolkit, Ilisys website and MySQL outages
Date: Saturday, 5 December 2009
Status: Complete

Summary:
On Saturday 5 December we’ll be conducting a series of scheduled maintenance activities that will require brief outages. A small number of customers will be affected, and the impacts are listed below. We apologise for any inconvenience caused as a result of these interruptions.

10.10am – 10.45am: Toolkit/website maintenance
Who is affected: Visitors to Ilisys.com.au and Ilisys Toolkit
Impact: No access to ilisys.com.au (including sales) and Ilisys Toolkit.

10:30 – 10:45am: Ilisys MySQL databases
Who is affected: Database driven sites (dynamic)
Impact: A small number of websites will be down briefly during this period.

Update: The scheduled system maintenance is now complete.

Posted in Scheduled | Comment »