2025-01-01 03:02:01.019970: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-01 03:02:01.034005: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-01 03:02:01.037988: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
INFO:mteb.cli:Running with parameters: Namespace(model='voyageai/voyage-3', task_types=None, categories=None, tasks=['RuBQReranking'], languages=None, benchmarks=None, device=None, output_folder='/data/niklas/results/results', verbosity=2, co2_tracker=True, eval_splits=None, model_revision=None, batch_size=64, overwrite=False, save_predictions=False, func=<function run at 0x7f912e3d9630>)
INFO:mteb.evaluation.MTEB:

Evaluating 1 tasks:

─────────────────────────────── Selected tasks ────────────────────────────────
Reranking
- RuBQReranking, s2p

INFO:mteb.evaluation.MTEB:

********************** Evaluating RuBQReranking **********************
No config specified, defaulting to the single config: rubq-reranking/default
INFO:datasets.builder:No config specified, defaulting to the single config: rubq-reranking/default
Loading Dataset Infos from /env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/packaged_modules/json
INFO:datasets.info:Loading Dataset Infos from /env/lib/conda/gritkto/lib/python3.10/site-packages/datasets/packaged_modules/json
Overwrite dataset info from restored data version if exists.
INFO:datasets.builder:Overwrite dataset info from restored data version if exists.
Loading Dataset info from /data/huggingface/datasets/ai-forever___rubq-reranking/default/0.0.0/2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2
INFO:datasets.info:Loading Dataset info from /data/huggingface/datasets/ai-forever___rubq-reranking/default/0.0.0/2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2
Found cached dataset rubq-reranking (/data/huggingface/datasets/ai-forever___rubq-reranking/default/0.0.0/2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2)
INFO:datasets.builder:Found cached dataset rubq-reranking (/data/huggingface/datasets/ai-forever___rubq-reranking/default/0.0.0/2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2)
Loading Dataset info from /data/huggingface/datasets/ai-forever___rubq-reranking/default/0.0.0/2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2
INFO:datasets.info:Loading Dataset info from /data/huggingface/datasets/ai-forever___rubq-reranking/default/0.0.0/2e96b8f098fa4b0950fc58eacadeb31c0d0c7fa2
INFO:mteb.abstasks.AbsTask:
Task: RuBQReranking, split: test, subset: default. Running...
INFO:mteb.evaluation.evaluators.RerankingEvaluator:Encoding queries...
INFO:mteb.evaluation.evaluators.RerankingEvaluator:Encoding candidates...
WARNING:mteb.evaluation.evaluators.RerankingEvaluator:A total on 2227/37447 duplicate texts were found during encoding. Only encoding unique text and duplicating embeddings across.
ERROR:mteb.evaluation.MTEB:Error while evaluating RuBQReranking: The request body is not valid JSON, or some arguments were not specified properly. In particular, Value 'passage' supplied for argument 'input_type' is not valid -- accepted values are 'query' or 'document'
Traceback (most recent call last):
File "/env/lib/conda/gritkto/bin/mteb", line 8, in
sys.exit(main())
File "/data/niklas/mteb/mteb/cli.py", line 387, in main
args.func(args)
File "/data/niklas/mteb/mteb/cli.py", line 145, in run
eval.run(
File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 623, in run
raise e
File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 576, in run
results, tick, tock = self._run_eval(
File "/data/niklas/mteb/mteb/evaluation/MTEB.py", line 304, in _run_eval
results = task.evaluate(
File "/data/niklas/mteb/mteb/abstasks/AbsTask.py", line 126, in evaluate
scores[hf_subset] = self._evaluate_subset(
File "/data/niklas/mteb/mteb/abstasks/AbsTaskReranking.py", line 89, in _evaluate_subset
scores = evaluator(model)
File "/data/niklas/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 70, in call
scores = self.compute_metrics(model)
File "/data/niklas/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 75, in compute_metrics
self.compute_metrics_batched(model)
File "/data/niklas/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 112, in compute_metrics_batched
results = self._encode_candidates(
File "/data/niklas/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 149, in _encode_candidates
self._encode_candidates_batched(
File "/data/niklas/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 179, in _encode_candidates_batched
all_docs_embs = self._encode_unique_texts(
File "/data/niklas/mteb/mteb/evaluation/evaluators/RerankingEvaluator.py", line 441, in _encode_unique_texts
model.encode(
File "/data/niklas/mteb/mteb/models/voyage_models.py", line 101, in encode
return self._batched_encode(sentences, batch_size, input_type)
File "/data/niklas/mteb/mteb/models/voyage_models.py", line 128, in _batched_encode
self._embed_func(
File "/data/niklas/mteb/mteb/models/voyage_models.py", line 58, in wrapper
result = func(*args, **kwargs)
File "/data/niklas/mteb/mteb/models/voyage_models.py", line 25, in wrapper
result = func(*args, **kwargs)
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/client.py", line 68, in embed
for attempt in self.retry_controller:
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/tenacity/init.py", line 435, in iter
do = self.iter(retry_state=retry_state)
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/tenacity/init.py", line 368, in iter
result = action(retry_state)
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/tenacity/init.py", line 390, in
self._add_action_func(lambda rs: rs.outcome.result())
File "/env/lib/conda/gritkto/lib/python3.10/concurrent/futures/_base.py", line 451, in result
return self.__get_result()
File "/env/lib/conda/gritkto/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/client.py", line 70, in embed
response = voyageai.Embedding.create(
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/api_resources/embedding.py", line 22, in create
response = super().create(*args, **kwargs)
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/api_resources/api_resource.py", line 47, in create
response = requestor.request(
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/api_resources/api_requestor.py", line 147, in request
resp = self._interpret_response(result)
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/api_resources/api_requestor.py", line 408, in _interpret_response
return self._interpret_response_line(
File "/env/lib/conda/gritkto/lib/python3.10/site-packages/voyageai/api_resources/api_requestor.py", line 463, in _interpret_response_line
raise self.handle_error_response(rbody, rcode, resp.data, rheaders)
voyageai.error.InvalidRequestError: The request body is not valid JSON, or some arguments were not specified properly. In particular, Value 'passage' supplied for argument 'input_type' is not valid -- accepted values are 'query' or 'document'
/var/spool/slurmd/job810583/slurm_script: line 9: /data/niklas/mteb/failures/voyageai/voyage-3_RuBQReranking.txt: No such file or directory