Skip to content

Expose rate limiting headers to responses and errors #39

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Oct 18, 2023
Merged

Expose rate limiting headers to responses and errors #39

merged 1 commit into from
Oct 18, 2023

Conversation

mrene
Copy link
Contributor

@mrene mrene commented Oct 17, 2023 •
edited
Loading

This exposes the rate limit information included inside response headers returned by OpenAI.

It performs a best effort to parse them and ignores errors, as we wouldn't want a call to fail due to failing to deserialize this information.

An example native way to consume these would look like:

var gpt3Err gpt3.APIError
if errors.As(err, &gpt3Err) {
	if gpt3Err.StatusCode == http.StatusTooManyRequests {
		if gpt3Err.RateLimitHeaders.RemainingRequests == 0 {
            // Wait until the the RPM limit resets
			time.Sleep(gpt3Err.RateLimitHeaders.ResetRequests)
		} else {
            // Wait until the the TPM limit resets
			time.Sleep(gpt3Err.RateLimitHeaders.ResetTokens)
		}
	}
}

pullrequest[bot]
pullrequest bot reviewed Oct 17, 2023
View reviewed changes
Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ This pull request was sent to the PullRequest network.

@mrene you can click here to see the review status or cancel the code review job.

pullrequest[bot]
pullrequest bot reviewed Oct 17, 2023
View reviewed changes
Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PullRequest Breakdown

Reviewable lines of change

+ 104
- 4

69% Go
31% Go (tests)

Type of change

Feature - These changes are adding a new feature or improvement to existing code.

pullrequest[bot]
pullrequest bot reviewed Oct 17, 2023
View reviewed changes
Copy link

@pullrequest pullrequest bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change looks great with a test case as the sample provided by openAI doc. The only comment I would consider is if we need any negative testcases in case some of the headers are not included from OpenAI response

Image of Xiaoyong W Xiaoyong W

Reviewed with ❤️ by PullRequest

tylermann
tylermann approved these changes Oct 18, 2023
View reviewed changes
Copy link
Contributor

@tylermann tylermann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for this change @mrene . I think the "best effort" approach makes sense to me and agree we don't want to fail the request from this.

I was trying to think if there was any other better way to surface these than adding to the response and error objects, but i think this is just as good as other ideas I have so lets go with it!

@tylermann tylermann merged commit f29b15e into PullRequestInc:main Oct 18, 2023
@mrene
Copy link
Contributor Author

mrene commented Oct 18, 2023

Thanks for the quick merges & releases!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
2 more reviewers

@pullrequest pullrequest[bot] pullrequest[bot] left review comments

@tylermann tylermann tylermann approved these changes

Reviewers whose approvals may not affect merge requirements
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@mrene @tylermann