Dear Customers, At 9:03PM on Friday Nov 23rd there was a power outage at our facility in Milano Caldera. This outage lasted until 9:05PM. As always, our UPS took over the computer room load waiting for the outage to end.
An Automatic Transfer Switch is used to start up the diesel generator and to switch the UPS power input from the grid to the generator. This ATS is protected by a dedicated UPS as well, being activated by an electric switch. Due to a faulty interlock that supervises the switch from grid to generator, the grid circuit was opened (closed) but the generator circuit was not closed (opened), even though the generator was already running. Throughout this timespan, UPS batteries sustained the load in place of the generator, which was therefore disconnected.
When the power retuned, two minutes after the outage started, the same interlock was unable to switch back the load on the grid circuit, therefore UPSs kept running the load on batteries. Meanwhile the battery charge ran low until the UPS shut off cutting the power to all 230v powered equipment in the computer room.
All systems hosted in Enter's Milano datacenter were unreachable until 10:16PM, when we could start rebooting all network equipment.
Yesterday afternoon all Enter and Customers systems were back to normal operation Meanwhile we are still investigating the root causes for the fault of a component which has never shown any problem before now, despite the monthly load tests we perform on the continuity systems.
Despite our continuing efforts to design and apply security and check procedures in conjunction with our constant and frequent maintenance facilities, we could not foresee the breakage of this component. Our engineers are working to evaluate, if possibile, to double the protection system with a secondary circuit to become operational in case of failure of the former. We deeply apologise for the inconvenience. Best regards, Enter Cloud Suite Team
Posted Nov 25, 2018 - 09:39 CET
Dear customers, we are finishing to restore our DC infrastructures and network connectivity. EnterCloudSuite was restore completeley and we are finiling the reboot of customer virtual machines.
Our core services are fully restored. Now we will proceed to check customer infrastructures.
Posted Nov 24, 2018 - 03:47 CET
Our block storage service is restored correctly. We are restoring our core services.
Posted Nov 24, 2018 - 00:24 CET
We are working hard to restore the power as safely and quickly as possible, your services will be restored as soon as possible.
Posted Nov 23, 2018 - 22:59 CET
We experienced a power outage in our Milan datacenter and we are running on our generators. We will provide additional information as soon as possible.
Posted Nov 23, 2018 - 22:37 CET
We're experiencing connectivity loss on the public side in the Milan region. We're investigating. We apologise for the inconvenience and we'll keep you posted as soon as we have completely resolved the issue. Enter Cloud Suite Team
Posted Nov 23, 2018 - 21:25 CET
This incident affected: O.Cloud IT-MIL1, Network, Hosting Services, and CloudUP / SelfServer.