We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation .
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement . We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
I have the following workflow configuration with subindices for two different datasets.
workflow.yaml
writable: true path: vector-database embeddings: content: true defaults: false indexes: document: path: sentence-transformers/multi-qa-mpnet-base-dot-v1 tokenize: true columns: text: document csv: path: sentence-transformers/multi-qa-mpnet-base-dot-v1 tokenize: true columns: text: csv
After adding the data, app.count() returns 1. However, when this data doesn't persist when the session is refreshed. app.count() returns 0 when run in another separate environment.
app.count()
from txtai.app import Application app = Application("search-workflow.yaml") app.add([{'document': 'dummy data 1', 'csv': 'dummy data 2'}]) app.upsert() app.count()
@davidmezzetti Could you please confirm if I am missing anything in the code and suggest what we could do to persist data?
The text was updated successfully, but these errors were encountered:
I'm using this search workflow as an intermediate step in another workflow in a RAG solution. The search workflow is not able to find the data in the vector database to retrieve relevant texts.
Sorry, something went wrong.
Sorry for the delayed response. I will try your config and let you know.
I ran the above configuration and it saves content to the vector-database directory.
vector-database
Can you share more on search workflow? Once you load an embeddings database it doesn't automatically refresh.You would need a way to reload the read-only search index when data is loaded. Could this be the issue?
Sorry for the delay. Yes, that seems to be the issue. Looks like it doesn't automatically load in read-only search index.
Ok, thank you for confirming. I can think about a method to autodetect index changes and force a refresh. I think that would be a good addition.
One idea in the meantime to think about is if you can trigger anything with the read-only process when you update your index.
Not sure if its the same thing, but with the same configuration above, i'm seeing situations where the index directories are not always created. I also noticed when I try to call index, it eventually errors. Should the index directories be added when txtai starts up or not until an index is called?
No branches or pull requests
