Skip to content

Add semantic chunking pipeline #812

@davidmezzetti

Description

@davidmezzetti

Evaluate using the Chonkie library to add more sophisticated chunking to txtai.

In reviewing the code, it appears that an interface wrapping a txtai vectors instance could be passed in. With this, any supported txtai vectors model (huggingface, sentence-transformers, LiteLLM, llama-cpp etc) could be used.

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions