Random Segmentation fault with embedding.search after upserting #498
Description
I'm getting segmentation fault "randomly" when using with FastApi. It usually happens after the index is updated.
3 example errors below when running some tests where the setup is:
- sqlite as db
- 10 embedding.search per second via GET (with
RLock A) embedding.upsert+embedding.persistfor ~2k docs tasking roughly 5 seconds to index. (withRLock A)- seg fault usually occurs in a couple dozen of embedding.search after the embedding update is done(e.g. 3-4 seconds after)
I'm a little stuck as the memory usage doesn't seem too bad with 800mb(its an existing index+db. about 400mb without loading the index). Any idea how I can debug this?
Thread 0x00000002996b3000 (most recent call first):
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 449 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 549 in feed_forward_chunk
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 236 in apply_chunking_to_forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 537 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 610 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 1020 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 108 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 128 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 75 in encode
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/vectors/transformers.py", line 48 in encode
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/vectors/base.py", line 136 in batchtransform
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 293 in batchtransform
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 71 in search
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 107 in dbsearch
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 53 in __call__
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 345 in batchsearch
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 329 in search
Thread 0x00000002a26b3000 (most recent call first):
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/activations.py", line 79 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 450 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 549 in feed_forward_chunk
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 236 in apply_chunking_to_forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 537 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 610 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 1020 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 108 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 128 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 75 in encode
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/vectors/transformers.py", line 48 in encode
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/vectors/base.py", line 136 in batchtransform
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 293 in batchtransform
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 71 in search
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 107 in dbsearch
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 53 in __call__
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 345 in batchsearch
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 329 in search
Thread 0x000000029fbb3000 (most recent call first):
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/linear.py", line 114 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 462 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 550 in feed_forward_chunk
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/pytorch_utils.py", line 236 in apply_chunking_to_forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 537 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 610 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 1020 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 108 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 128 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/models/pooling.py", line 75 in encode
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/vectors/transformers.py", line 48 in encode
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/vectors/base.py", line 136 in batchtransform
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 293 in batchtransform
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 71 in search
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 107 in dbsearch
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/search.py", line 53 in __call__
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 345 in batchsearch
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/txtai/embeddings/base.py", line 329 in search
Thread 0x0000000297f3f000 (most recent call first):
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/functional.py", line 2515 in layer_norm
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/normalization.py", line 190 in forward
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1501 in _call_impl
File ".../.pyenv/versions/3.11.1/lib/python3.11/site-packages/transformers/models/bert/modeling_bert.py", line 465 in forward