Skip to content

[mieb] kakaobrain/align-base fails on ImageNet10Clustering #1387

@Muennighoff

Description

@Muennighoff
ERROR:mteb.evaluation.MTEB:Error while evaluating ImageNet10Clustering: Unsupported number of image dimension
s: 2                                                                                                         
Traceback (most recent call last):
  File "/data/niklas/mieb/mteb/scripts/run_mieb.py", line 83, in <module>          
    results = evaluation.run(model, output_folder="/data/niklas/mieb/results-mieb-final", batch_size=1)      
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 464, in run                                  
    raise e                                                                                                  
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 425, in run                                    
    results, tick, tock = self._run_eval(         
  File "/data/niklas/mieb/mteb/mteb/evaluation/MTEB.py", line 300, in _run_eval              
    results = task.evaluate(                                                                                 
  File "/data/niklas/mieb/mteb/mteb/abstasks/AbsTask.py", line 126, in evaluate                              
    scores[hf_subset] = self._evaluate_subset(                                                               
  File "/data/niklas/mieb/mteb/mteb/abstasks/Image/AbsTaskImageClustering.py", line 54, in _evaluate_subset  
    metrics = evaluator(model, encode_kwargs=encode_kwargs)                                                  
  File "/data/niklas/mieb/mteb/mteb/evaluation/evaluators/Image/ClusteringEvaluator.py", line 41, in __call__
    image_embeddings = model.get_image_embeddings(   
  File "/data/niklas/mieb/mteb/mteb/models/align_models.py", line 60, in get_image_embeddings                
    inputs = self.processor(                                                                                 
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/align/processing_align.py", 
line 126, in __call__                                                                                        
    image_features = self.image_processor(images, **output_kwargs["images_kwargs"])
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_processing_utils.py", line 41
, in __call__                                                                                                
    return self.preprocess(images, **kwargs)
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/utils/generic.py", line 852, in wra
pper                                                  
    return func(*args, **valid_kwargs)                                                                       
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/models/efficientnet/image_processin
g_efficientnet.py", line 328, in preprocess                                                                  
    input_data_format = infer_channel_dimension_format(images[0])
  File "/env/lib/conda/gritkto4/lib/python3.10/site-packages/transformers/image_utils.py", line 243, in infer
_channel_dimension_format
    raise ValueError(f"Unsupported number of image dimensions: {image.ndim}")
ValueError: Unsupported number of image dimensions: 2

Metadata

Metadata

Assignees

No one assigned

    Labels

    miebThe image extension of MTEB

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions