Infoplus System Outage

Incident Report for InfoPlus

Postmortem

On March 15th, there was a failed database cluster process where only one of our two instances in the cluster was receiving connections. This persisted throughout the evening on the 15th and into the 16th, when we had system-wide slow processing speeds.

Today, our system reinstated the connection to all instances in the database cluster and during that time our engineers restarted all systems to refresh the database connection, which resulted in the system-wide outage from 1140am - 1240pm CST.

We are currently setting up new alarm points to alert the Development Team of any potential for failure in the database cluster.

All systems are now operational. If you continue to experience downtime or slower processing times, please contact Infoplus Support.

Posted Mar 17, 2022 - 16:27 CDT

Resolved

This incident has been resolved.

Posted Mar 17, 2022 - 16:27 CDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Mar 17, 2022 - 13:27 CDT

Update

We are continuing to investigate this issue.

Posted Mar 17, 2022 - 12:29 CDT

Update

Infoplus engineers have restarted the database cluster connection but system speeds will still be slow as queues catch up.

Posted Mar 17, 2022 - 12:25 CDT

Update

We have discovered that a database failed to reconnect during a failover into its' cluster. Our engineers are actively working to re-stabilize the system and reapply load balancing. Please expect slower-than-usual processing times as we apply our changes.

Posted Mar 17, 2022 - 11:22 CDT

Investigating

Infoplus is experiencing a partial outage impacting our desktop, mobile, and API services. Expect slower-than-usual processing times as we investigate and address the root issue.

We will provide an update as we learn more.

Posted Mar 17, 2022 - 10:50 CDT

This incident affected: Infoplus Web Application, Infoplus API, and Infoplus Mobile Floor Apps.