Summary of Events: Initial Notification: In January, AWS announced a mandatory upgrade requirement for Infoplus from their current MySQL database to the most recent version, to ensure continued performance and security compliance.
Scheduled Upgrade Plan: The Infoplus Development team scheduled the upgrade for Sunday, February 25th, with a maintenance window from 12:00 PM to 5:00 PM. This upgrade addressed the changes enforced by AWS and maintained optimal system performance.
Upgrade Process: QA Environment Upgrade: Initially, the team upgraded the Quality Assurance (QA) environment to identify and resolve any potential issues before deploying changes to the production environment. Production Upgrade Attempt: Following successful QA testing, the team attempted to push the update to the production environment. Issues Encountered: The production upgrade was unexpectedly blocked due to unforeseen issues, preventing the completion of the upgrade as planned.
Impact on Services: Order Processing Speed: There is a noticeable slowdown in the processing of orders, affecting operational efficiency. Data Interruption Errors: Users may experience intermittent errors impacting data reliability and access, particularly non-ASCII characters like emojis. Audit Log Delays: The generation and integration of audit logs into the system are experiencing general slowness, affecting monitoring and tracking capabilities. All logs are still in queue and being added as quickly as possible. Timezone Display Issues: Users will notice timestamps displayed in UTC instead of their local timezone. Bulk Upload: Bulk upload is experiencing difficulties with the drag and drop / browse for file.
Current Status: Operational with Limitations: The system is operational but experiencing slower processing speeds than usual. The Infoplus Development team is actively addressing the issues caused by the attempted upgrade to minimize disruption and improve system performance. Ongoing Resolution Efforts: We are committed to resolving the identified issues as quickly and efficiently as possible. Our team is focused on optimizing system operations and ensuring stability across all services. Updates and Communication: We will continue to provide timely updates regarding our progress and any changes to the system status. Your patience and understanding during this time are greatly appreciated.
Recommendations for Users: Monitor Status Updates: Please continue to check our status page for the most current information and developments related to these issues. Contact Support: For urgent issues or further assistance, our support team is available to help address your concerns and provide support.
Infoplus is currently working to resolve the MySQL errors and updating the necessary tables to proceed with the update. Additionally, new job servers are being configured to be added to the job processing pool to help increase the processing speeds.
Posted Feb 28, 2024 - 14:22 CST
This incident affected: Infoplus Web Application, Infoplus API, Infoplus Support Site, and Infoplus Mobile Floor Apps.