Initial report and findings

April 18, 2024

Incident date and time
Date: 4/18/24
Time: Approximately 10:20 AM EST to 10:35AM EST

Incident summary

Users of the iPOSpays gateway were unable to process transactions or access the gateway portal. The duration of this incident was approximately 15 minutes.

Summary of issue

At approximately 10:20 AM EST the database servers for the iPOSpays gateway became unresponsive due to a suspected network issue at AWS. Our alerting and monitoring system detected the issue immediately. Once the issue was identified our systems team pointed all the applications to our Disaster Recovery instance. Our Disaster Recovery instance is set up at full capacity as the primary instance.

The Disaster Recovery instance was activated and live within 15 minutes, this is within our published TAT for bringing up DR instance.

Our entire application and global dev-ops resources coordinated and executed the DR cutover plan as per previously established within the Standard Operating Procedure (SOP).

Downtime is approximately between 10.20am EST and 10.35am EST

Root cause analysis

Through our Initial analysis it is suspected that the root cause of this issue is due to a network issue at AWS. A complete and thorough investigation is pending.

Response handling
• At 10:20 EST the problem was detected by internal monitoring systems, which initiated the activation of the     Disaster Recovery Plan.
• The Disaster Recovery instance of the gateway was activated.
• At approximately 10:35 transactions were able to be processed, and Ipospays portal was restored.

Remedial steps
• Complete full analysis of the incident, to fully understand root cause issue.

Download the press release here.