Addressing Potential System Failures and Implementing Redundancy Measures with Redis
The State Changers discussed the risk of dependency on RapidAPI for their game infrastructure, which is largely feed-based. In the event RapidAPI becomes unavailable, they have created a workaround to keep their system operational by being able to generate games based off cached data. However, there are concerns about the data’s longevity and quality in this case, alongside worries about potential accidental deletion of the cache.
The consensus is to implement safeguards such as creating a backup of the data, possibly through a background task or function endpoint, which would push a JSON to a third-party logging service or a database. The concept of creating redundancy is discussed as a means to safeguard the data.
The discussion also highlights the importance of having a secondary vendor as a backup, which could take over if the primary one fails. This redundant vendor should ideally be set up to an extent that it can quickly take over operations to minimize downtime.
Despite having a backup plan for maintaining operations in the first 48 hours (roughly the longevity of data freshness), they acknowledge the need to plan for the scenario post this period. They recognize that the larger problem is what to do when the data goes beyond this freshness period, and the quality of games starts deteriorating.
Another aspect of the meeting revolved around the time it takes to back up data. Currently, they only cover three sports, and the full backup requires around eight minutes, mainly due to the 7-8 seconds response time of RapidAPI. They discuss the necessity to frequently update data, and yet the inefficiency it implies in terms of overall backup time.