Skip to content

Implement rate limiting in all API models #1516

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Jul 4, 2025
Merged

Implement rate limiting in all API models #1516

merged 5 commits into from
Jul 4, 2025

Conversation

albertvillanova
Copy link
Member

Implement rate limiting in all API models.

This PR implements rate limiting across all API model classes by adding calls to self._apply_rate_limit() before API requests are made.

Changes:

  • Implement RateLimiter class
  • Added RateLimiter class initialization in ApiModel.init
  • Implemented _apply_rate_limit() method in ApiModel
  • Added rate limiting calls in all API model implementations:
    • LiteLLMModel
    • InferenceClientModel
    • OpenAIServerModel
    • AzureOpenAIServerModel
    • AmazonBedrockServerModel
    • LiteLLMRouterModel
  • Added calls in both generate() and generate_stream() methods

Follow-up to:

@aymeric-roucher
Copy link
Collaborator

It seems to me that common providers lit rate limits more in Requests Per Minute (RPM), cf Nebius, Anthropic...: maybe implement this per minute instead of per second?

aymeric-roucher
aymeric-roucher approved these changes Jul 4, 2025
View reviewed changes
Copy link
Collaborator

@aymeric-roucher aymeric-roucher left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you! It would be nice to add a mention of this possibility to rate limit in the docs and add the arg in some of the examples!

@HuggingFaceDocBuilderDev
Copy link

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@albertvillanova albertvillanova merged commit 596af22 into huggingface:main Jul 4, 2025
4 checks passed
@albertvillanova albertvillanova deleted the rate-limit-model branch July 4, 2025 10:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@aymeric-roucher aymeric-roucher aymeric-roucher approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@albertvillanova @aymeric-roucher @HuggingFaceDocBuilderDev