Skip to content

HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1 fails #1657

@Muennighoff

Description

@Muennighoff
2025-01-01 04:07:12.442556: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:485] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2025-01-01 04:07:12.456047: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:8454] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2025-01-01 04:07:12.459781: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1452] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
INFO:mteb.cli:Running with parameters: Namespace(model='HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1', task_types=None, categories=None, tasks=['BiorxivClusteringP2P.v2'], languages=None, benchmarks=None, device=None, output_folder='/data/niklas/results/results', verbosity=2, co2_tracker=True, eval_splits=None, model_revision=None, batch_size=32, overwrite=False, save_predictions=False, func=<function run at 0x7fbe7ae68f70>)
WARNING:mteb.model_meta:Loader not specified for model HIT-TMG/KaLM-embedding-multilingual-mini-instruct-v1, loading using sentence transformers.
Traceback (most recent call last):
  File "/env/lib/conda/gritkto/bin/mteb", line 8, in <module>
    sys.exit(main())
  File "/data/niklas/mteb/mteb/cli.py", line 387, in main
    args.func(args)
  File "/data/niklas/mteb/mteb/cli.py", line 123, in run
    model = mteb.get_model(args.model, args.model_revision, device=device, trust_remote_code=True)
  File "/data/niklas/mteb/mteb/models/overview.py", line 150, in get_model
    model = meta.load_model(**kwargs)
  File "/data/niklas/mteb/mteb/model_meta.py", line 128, in load_model
    model: Encoder = loader(**kwargs)  # type: ignore
  File "/data/niklas/mteb/mteb/model_meta.py", line 40, in sentence_transformers_loader
    return SentenceTransformerWrapper(model=model_name, revision=revision, **kwargs)
  File "/data/niklas/mteb/mteb/models/sentence_transformer_wrapper.py", line 48, in __init__
    model_prompts = self.validate_task_to_prompt_name(self.model.prompts)
  File "/data/niklas/mteb/mteb/models/wrapper.py", line 83, in validate_task_to_prompt_name
    task = mteb.get_task(task_name=task_name)
  File "/data/niklas/mteb/mteb/overview.py", line 318, in get_task
    raise KeyError(suggestion)
KeyError: "KeyError: 'document' not found and no similar keys were found."

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions