Multiple references for meteor is still not working.

>>> meteor = evaluate.load('meteor')
>>> predictions = ["It is a guide to action which ensures that the military always obeys the commands of the party"]
>>> references = [['It is a guide to action that ensures that the military will forever heed Party commands', 'It is the guiding principle which guarantees the military forces always being under the command of the Party', 'It is the practical guide for the army always to heed the directions of the party']]
>>> results = meteor.compute(predictions=predictions, references=references)
>>> print(round(results['meteor'], 2))

results in this stack error:

[nltk_data] Downloading package wordnet to
[nltk_data]     /home/vishalburman/nltk_data...
[nltk_data]   Package wordnet is already up-to-date!
[nltk_data] Downloading package punkt to
[nltk_data]     /home/vishalburman/nltk_data...
[nltk_data]   Package punkt is already up-to-date!
[nltk_data] Downloading package omw-1.4 to
[nltk_data]     /home/vishalburman/nltk_data...
[nltk_data]   Package omw-1.4 is already up-to-date!
Traceback (most recent call last):
  File "sample.py", line 6, in <module>
    results = meteor.compute(predictions=predictions, references=references)
  File "/home/vishalburman/miniconda3/envs/eval/lib/python3.8/site-packages/evaluate/module.py", line 425, in compute
    self.add_batch(**inputs)
  File "/home/vishalburman/miniconda3/envs/eval/lib/python3.8/site-packages/evaluate/module.py", line 503, in add_batch
    raise ValueError(error_msg) from None
ValueError: Module inputs don't match the expected format.
Expected format: {'predictions': Value(dtype='string', id='sequence'), 'references': Value(dtype='string', id='sequence')},
Input predictions: ['It is a guide to action which ensures that the military always obeys the commands of the party'],
Input references: [['It is a guide to action that ensures that the military will forever heed Party commands', 'It is the guiding principle which guarantees the military forces always being under the command of the Party', 'It is the practical guide for the army always to heed the directions of the party']]

Rouge and Meteor for multiple references #118

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions