The State Changers discussed an email and webhook situation during this meeting, with emphasis on reconciliation and potential points of failure as the scale of data increases. Using platforms like "Xano", they talked about developing an efficient system to manage and reconcile data from emails.
They introduced a Boolean field "reconciled" that checks whether data have been reconciled, with a background task that queries unreconciled records and processes them, marking the data as reconciled once completed. The primary concern discussed was ensuring accuracy even as the data volume increases significantly. The meeting explored potential solutions and improvements, noting opportunities to speed up the process in the future. These include introducing states such as 'queued', 'running', 'done', and 'error' rather than just indicating the data as reconciled. This creates more room to understand where the process stands at any given point and helps in troubleshooting and finding 'problem children' by searching for 'running' processes older than a specific time. Concerns were raised about potential points of failure with increasing volumes of data, like faults with webhook calls, reconciliation failure, and correct aggregation. The State Changers agreed that not all points are equally sensitive to scale. Suggestions were made to possibly seek help from services like 'HookDeck' to handle webhooks, especially when the data volume increases rapidly. The advantage of third-party services would be their ability to retain data for longer (e.g., seven days compared to Xano's twenty-four hours), providing an added layer of data security. The trade-off between the cost of third-party services and the time spent reengineering things was discussed, with greater priority given to adapting the system for high-value customers effectively. The meeting concluded with the plan to revisit another question at the centre of the discussion depending on how much time remained.
(Source: Office Hours 2/17 )
