Addressing Memory Management and XML Parsing Issues When Handling High Volume Data with Xano and EasyLambda
In this meeting, the State Changers had a robust discussion centered around problems they were encountering with parsing large XML data feeds using Xano. With the limit of memory and the nature of the platform favouring smaller, quick tasks, the participants doubted if Xano was the suitable tool for handling large XML feeds.
They highlighted how Xano is designed to quickly handle many small, fast operations (these operations forming the 'cross of the T' in a 'T-shaped problem') but struggles with larger scale tasks. Due to this, they found themselves creating Lambda functions as a work-around for the limitations encountered with Xano.
The meeting then delved into a live debugging session trying to dissect the issues of requesting and handling XML data. They utilized console logs, running tests, and examining request history in real time to troubleshoot the problem. The process highlighted the challenges of debugging in a no-code/low-code environment.
Ultimately, they successfully manipulated the data into smaller JSON chunks using "EasyLambda" (which they considered renaming to "HeavyLambda"). They noted this solution may not be the fastest but would be reliable for processing tasks that were not time-critical to user experience, like nightly data crunching tasks.
Keywords mentioned: Xano, Lambda, serverless, State Change.