Addressing Terminal Freezing

Introduction:
On December of 2019, we encountered a sporadic but persistent issues with our Linux terminals freezing during the “System Initializing” phase. In response, we initiated a thorough investigation and collaborated with our manufacturer, Castles Technology, to identify and resolve the root cause of the problem. This issue led to temporary disablement of terminals and the possibility of memory not being fully recoverable or operable.

Problem Identification:
The root cause of the problem was attributed to corruption in the Linux filesystem, which became apparent during terminal startup or reboot processes. The issue was traced back to the download process, where temporary file remnants left behind during file extraction consumed terminal memory over time. This led to application initialization failures as the available memory became insufficient.

Collaborative Resolution:
We worked closely with Castles Technology to devise a solution. This involved implementing a “memory clean” and “memory check” before each download to prevent the accumulation of temporary files. Additionally, we revamped the download process to execute sequential file downloads, reducing memory requirements and eliminating leftover remnants.

Mitigation Measures:
To minimize disruptions while the solution was being developed, we temporarily disabled the auto-update feature and partial downloads. This reduced the frequency of occurrences until a permanent fix could be implemented.

We offered a temporary solution as well with existing software, terminals were sent to us for a SULD (Memory Reset) operation before being returned to the field.

Testing and Quality Assurance:
The updated software underwent rigorous quality assurance testing, including selective tests in production environments. Successful testing ensured that the solution was robust and reliable.

Rollout Strategy:
Before the general release, we coordinated with customers to ensure a smooth rollout. Incremental deployment allowed for careful monitoring and ensured minimal impact on the install base.

Customer Support and Assistance:
Throughout the process, we prioritized customer satisfaction. We offered free terminals or replacements to affected customers and provided prompt assistance through our technical support, relationship management, and sales teams.

Conclusion:
Through collaboration, thorough investigation, and proactive measures, we successfully resolved the terminal freezing issue. Our commitment to customer satisfaction and efficient problem-solving demonstrates our dedication to providing reliable and flexible software solutions in the payment processing industry.