Skip to content

[v2] reupload reranking datasets in old format #2097

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 13 commits into from
Feb 24, 2025
Merged

[v2] reupload reranking datasets in old format #2097

merged 13 commits into from
Feb 24, 2025

Conversation

Samoed
Copy link
Member

@Samoed Samoed commented Feb 18, 2025 •
edited
Loading

Closes #1878

  • "MindSmallReranking",
  • "SciDocsRR",
  • "StackOverflowDupQuestions",
  • "WebLINXCandidatesReranking",
  • "AlloprofReranking",
  • "SyntecReranking",
  • "VoyageMMarcoReranking",
  • "ESCIReranking",
  • "MIRACLReranking",
  • "WikipediaRerankingMultilingual",
  • "RuBQReranking",
  • "T2Reranking",
  • "MMarcoReranking",
  • "CMedQAv1-reranking",
  • "CMedQAv2-reranking",
  • "NamaaMrTydiReranking",

Code Quality

  • Code Formatted: Format the code using make lint to maintain consistent style.

Documentation

  • Updated Documentation: Add or update documentation to reflect the changes introduced in this PR.

Testing

  • New Tests Added: Write tests to cover new functionality. Validate with make test-with-coverage.
  • Tests Passed: Run tests locally using make test or make test-with-coverage to ensure no existing functionality is broken.

@orionw Can you review, please?

@Samoed Samoed added the v2 Issues and PRs related to `v2` branch label Feb 18, 2025
@Samoed Samoed marked this pull request as ready for review February 20, 2025 15:22
@Samoed Samoed requested a review from orionw February 20, 2025 15:22
@Samoed Samoed mentioned this pull request Feb 20, 2025
Merged
4 tasks
@Samoed Samoed mentioned this pull request Feb 21, 2025
Merged
4 tasks
orionw
orionw approved these changes Feb 21, 2025
View reviewed changes
Copy link
Contributor

@orionw orionw left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great @Samoed, thanks for finalizing this.

My only question is does this affect the listing of tasks? e.g. what distinguishes reranking vs retrieval now, when you load with the mteb.load_tasks for reranking only? I would assume we need some mapping now to maintain that they are reranking tasks.

EDIT: perhaps that's in the task meta? If so then I have no concerns.

@Samoed
Copy link
Member Author

Samoed commented Feb 21, 2025

These tasks will still have type="Reranking" in their TaskMetadata, so this won’t break anything. However, for now, I think I’ll roll back the RerankingTask deletion since there might be more reranking tasks from main that need to be uploaded. I’ll wait for #2116 to be merged, as it includes a new reranking task.

@Samoed Samoed merged commit 8c27b80 into v2.0.0 Feb 24, 2025
3 of 8 checks passed
@Samoed Samoed deleted the reupload_retrieval_datasets branch February 24, 2025 14:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@orionw orionw orionw approved these changes

@KennethEnevoldsen KennethEnevoldsen Awaiting requested review from KennethEnevoldsen

Assignees
No one assigned
Labels
v2 Issues and PRs related to `v2` branch
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@Samoed @orionw