This meeting discussed a critical database management issue State Changers were having with their Xano-based application. They used to store files as base 64 inside of text fields, effectively creating a custom file management solution. However, the large file sizes were causing database failures because of foreign key relationships that pulled data from one table to another.
One key solution implemented to address the issue was increasing the memory, which allowed the database to function properly. However, due to the nature of the application—an auditing system—they raised the concern about logger reliability and ensuring that all added rows were successfully registered. State Changers suggested using the metadata API to implement a logging system. They could iterate through each instance, getting the recent request history and saving it as a JSON object to start with. The main goal was to have reliable storage of data first, and then look for options for efficiency and scalability later. There were thoughts about outsourcing logger services to dedicated platforms like Datadog. However, considerations about data integrity and learning from the building experience led to the decision that, at least for the time being, they would build the logging system in-house. This would also give them a proprietorial tool, which could be an important asset. The meeting wrapped up with them wanting to learn from their mistakes and use them to gear up for bigger challenges in the future. Themes such as data management, integrity, and reliability were the main focus points of this discussion. This meeting would be highly valuable for anybody dealing with similar issues or wanting to explore efficient solutions for data logging and retrieval systems.
(Source: Office Hours 6/21 )
