In the meeting, the State Changers discussed issues around using Flowise to scrape and upload data to Pinecone via Xano. A participant encountered a token limit error because the chunks of data being attempted to upload were too large for OpenAI embedding limitations. As a solution, it was suggested that these chunks be divided further and uploaded individually into Pinecone for semantic search. Lang chain structure was proposed for creating the embeddings.
Additionally, the role of a batcher or splitter was underlined for processing large amounts of data, like the documents of docs.xano.com. The need to add it in the chain between the Cheerio web scraper and Pinecone upsert document was identified. The conversation also touched upon the topic of deploying Flowise on Railway app, where redeployment wiped out the entire memory. A security concern with Railway was also raised. The participant was advised to post about these individual issues on the forum for further discussion and resolution. The meeting ended on a hopeful note, with the promise of continuing the conversation with added inputs from the posts.
(Source: Office Hours 7/7/2023 )
