Singapore’s Senior Minister of State for Health Janil Puthucheary said that the two IT system outages at healthcare institutions recently were caused by failures of hardware devices in data centres, not a lack of manpower.
Dr Puthucheary was responding to questions by MPs in Parliament on Monday (Sep 12) on the two outages which occurred on Aug 27 and Sep 5, according to a report by Channel News Asia. Member of Parliament (MP) Tan Wu Meng (PAP-Jurong GRC) asked for a breakdown of the patients affected by the outage, its causes, and how the Integrated Health Information Systems (IHiS) compared against international healthcare institutions, while MP Wan Rizal Wan Zakariah (PAP-Jalan Besar) asked for safeguards in place to prevent similar disruptions in the future.
Failure of IT Vendors?
Industry watchers opined that this development clearly highlights vendors’ inability to meet the demanding needs in a data center. Detailing the incidents, Dr. Puthucheary said that both incidents had caused a ‘significant impact on operations’. Due to the outages, patients had longer waiting times of up to an hour at affected institutions, others had appointments rescheduled and there were also delays in dispensing medication.
“On both 27th August and 5th September, our affected public healthcare institutions activated their downtime procedures and business continuity plans to keep operations running using alternative systems and in some cases manual documentation,” he said.
In the first incident on August 27, the public healthcare monitoring systems detected IT network connectivity failures from 7am. The faults were rectified and systems restored by 10.45 am the same day. In total, 26 IT applications, including electronic medical records, appointment, pharmacy and laboratory systems, were affected, according to the CNA report.
A total of 17 public healthcare institutions, including community hospitals, specialist outpatient clinics, and all polyclinics were impacted. In the second incident on Sep 5, another fault occurred in the IT infrastructure at 10am. Partial functionality was restored from 1pm that day, while full functionality was only restored by 6pm the next day.
The disruption affected eight public healthcare institutions and two out of three polyclinic groups, according to Dr. Puthucheary. Investigations into the incidents showed that the main causes were failures of hardware devices in data centers, said Dr Puthucheary. In the days leading up to the Aug 27 incident, two nodes – hardware devices which operate in tandem – in firewall zones at data centers failed.
The system usually has other nodes to manage the load of data traffic to continue service operations if one fails. On Aug 27, as engineers tried to restore the two failed nodes, the operation failed.
Remedial Measures
This caused the cluster of firewall nodes to malfunction, which subsequently caused the outage. Dr Puthucheary said that the engineers worked to reset the systems to the prior state without the function of the two affected nodes and service was progressively restored.
“The failure of the nodes was caused by bugs in the firmware of the devices. They have since been identified by the manufacturer, Cisco, and the devices have been patched,” he added.
The Sep 5 incident was caused by “the simultaneous failure of two further nodes, again from the same manufacturer, and of the same model”, noted Dr Puthucheary. The way the failure occurred was different from the previous incident and more time was needed to restore operations. The cause of this failure is still under investigation.
“There was a suggestion in one of questions from members that the failures may be due to the lack of manpower of IHiS. IHiS has a headcount of 3,500 personnel, they have a lot to do and will always welcome more manpower, but a lack of manpower is not the cause of these failures,” said the Senior Minister of State.
He added that there were no indications of security compromise to the affected systems based on investigations.
