Skip to content

Handling truncation for the Similarity pipeline #882

@ccdv-ai

Description

@ccdv-ai

I have a reranker (cross encoder) which relies on an absolute positional embedding, limited to 8192 tokens.
In a situation where the length of prompt + document > 8192 tokens, it stops.
Tried using kwargs but didnt find the one passed to the tokenizer

result = self.pipeline([{"text": q, "text_pair": t} for t in texts], top_k=None, function_to_apply="none", num_workers=workers)

tokenizer = AutoTokenizer.from_pretrained(reranker_path)
model = AutoModelForSequenceClassification.from_pretrained(reranker_path, torch_dtype=torch.float16)
similarity_scorer = Similarity(
	(model, tokenizer), 
	crossencode=True, 
)

How to truncate on the right?

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions