Skip to content

model: add kalm_models (kalm-emb-v2) ModelMeta (new PR) #2889

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
Jul 15, 2025
Merged

model: add kalm_models (kalm-emb-v2) ModelMeta (new PR) #2889

merged 11 commits into from
Jul 15, 2025

Conversation

ItsukiFujii
Copy link
Contributor

If you add a model or a dataset, please add the corresponding checklist:

  • I have filled out the ModelMeta object to the extent possible
  • I have ensured that my model can be loaded using
    • mteb.get_model(model_name, revision) and
    • mteb.get_model_meta(model_name, revision)
  • I have tested the implementation works on a representative set of tasks.
  • The model is public, i.e. is available either as an API or the wieght are publicly avaiable to download

@ItsukiFujii ItsukiFujii mentioned this pull request Jul 9, 2025
Merged
6 tasks
Samoed
Samoed reviewed Jul 9, 2025
View reviewed changes
Samoed
Samoed reviewed Jul 10, 2025
View reviewed changes
mteb/models/kalm_models.py Outdated
Comment on lines 773 to 777
def kalmv2_instruct_loader(model_name_or_path, **kwargs):
model = InstructSentenceTransformerWrapper(
model_name_or_path,
**kwargs,
)
return model
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you use InstructSentenceTransformerWrapper direcly in ModelMeta?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ItsukiFujii Can you remove this function?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Samoed
Function kalmv2_instruct_loader has been removed

@ItsukiFujii
Copy link
Contributor Author

Hi @Samoed
I'm confused about why this error occurred in the code. If you have any ideas, please let me know

mteb/evaluation/MTEB.py:672: in run
    raise e
mteb/evaluation/MTEB.py:625: in run
    results, tick, tock = self._run_eval(
mteb/evaluation/MTEB.py:307: in _run_eval
    results = task.evaluate(
mteb/abstasks/AbsTaskMultilabelClassification.py:159: in evaluate
    scores[hf_subset] = self._evaluate_subset(
mteb/abstasks/AbsTaskMultilabelClassification.py:194: in _evaluate_subset
    sample_indices, _ = self._undersample_data_indices(
mteb/abstasks/AbsTaskMultilabelClassification.py:256: in _undersample_data_indices
    if any((label_counter[label] < samples_per_label) for label in y[i]):
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/datasets/arrow_dataset.py:669: in __getitem__
    return self.source._fast_select_column(self.column_name)[key][self.column_name]
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/datasets/arrow_dataset.py:2859: in __getitem__
    return self._getitem(key)
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/datasets/arrow_dataset.py:2841: in _getitem
    formatted_output = format_table(
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/datasets/formatting/formatting.py:654: in format_table
    query_type = key_to_query_type(key)
/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/datasets/formatting/formatting.py:574: in key_to_query_type
    _raise_bad_key_type(key)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

key = np.int64(0)

    def _raise_bad_key_type(key: Any):
>       raise TypeError(
            f"Wrong key type: '{key}' of type '{type(key)}'. Expected one of int, slice, range, str or Iterable."
        )
E       TypeError: Wrong key type: '0' of type '<class 'numpy.int64'>'. Expected one of int, slice, range, str or Iterable.

/opt/hostedtoolcache/Python/3.9.23/x64/lib/python3.9/site-packages/datasets/formatting/formatting.py:44: TypeError

@Samoed
Copy link
Member

Samoed commented Jul 11, 2025

Yes, this is problem with datasets. I've fixed it. Can you update your branch with latest main?

@ItsukiFujii
Copy link
Contributor Author

Yes, this is problem with datasets. I've fixed it. Can you update your branch with latest main?

Okay, thank you very much

@ItsukiFujii
Copy link
Contributor Author

Hi @Samoed
All checks have passed. It would be nice if you could please review this PR :)

@Samoed Samoed changed the title model: add kalm_models (kalm-emb-v2, kalm-x) ModelMeta (new PR) model: add kalm_models (kalm-emb-v2) ModelMeta (new PR) Jul 15, 2025
@Samoed Samoed merged commit 9ecac21 into embeddings-benchmark:main Jul 15, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Samoed Samoed Samoed left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ItsukiFujii @Samoed @YanshekWoo