REST API for "List repository activities" not paginating #73014

nasuoa · 2023-10-31T05:06:27Z

nasuoa
Oct 31, 2023

GitHub REST API

I'm trying to use this endpoint:

GET /repos/{owner}/{repo}/activity

https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#list-repository-activities

Why is there no page query parameter? There's a per_page parameter, but that's not enough to traverse the entire set of activities. I have the code that loops over the pages, just like every other endpoint that works with paginating. But, it ends up being an infinite loop as it never exhausts the full list of activities since the "page" parameter is completely ignored (the most likely situation, as it doesn't appear in the docs as a valid query parameter when it most-definitely should).

Am I missing something? The only alternative I can think of, would be to manually paginate using before and after query parameters, but that will require a lot of extra logic that defeats the purpose (especially would need to write logic to avoid duplicating, etc). That's one nightmare I prefer to avoid :-)

I would like to note, that it works perfectly well when the "per_page" value is more than enough for the full set of activities. But, the maximum allowed value for "per_page" is 100, and I have repos that have many more activities. So I definitely need the next pages...

One other intermediate solution is to use the activity_type and get 100 of each of push, force_push, etc,. But, again, there would be plenty more than 100 of the "push" type.

Thanks heaps.

Answered by b8zeek

Nov 6, 2023

No worries buddy. ☺️ I am glad you found the solution.

Regarding cursor based pagination, here's an example of the headers I got when I was testing that page thing you were speaking about:

{
  'access-control-allow-origin': '*',
  'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
  'cache-control': 'private, max-age=60, s-maxage=60',
  'content-encoding': 'gzip',
  'content-security-policy': "default-src 'none'",
  'content…

View full answer

ghostinhershell · 2023-11-06T05:17:55Z

ghostinhershell
Nov 6, 2023
Maintainer

Hey there @nasuoa 👋🏾

Thanks so much for reaching out in our community and welcome! We actually had a community member submit a how to on using pagination with GitHub's API to the community that I'm really excited to recommend to you. You can find that here: How To Use Pagination With GitHub's API.

This more than likely will answer your question but, if you do have any additional questions or would like clarification, please feel free to follow-up and let me know!

1 reply

nasuoa Nov 6, 2023
Author

Hi @ghostinhershell

Thank you for the follow up. I'm familiar with how pagination works (I've used it for plenty of the other GitHub API endpoints). The issue is that this particular endpoint I am using seems incomplete.

There is no option to specify the page parameter. The API ignore this parameter (and it's also not listed in the documentation). So, this leads me to believe it is an oversight and somehow it was completely missed.

b8zeek · 2023-11-06T06:28:30Z

b8zeek
Nov 6, 2023

This is definitely really interesting... 😅

Obviously, GET /repos/{owner}/{repo}/activity uses cursor based pagination and there is no option to navigate with page... It is in the documentation! I just tested the API, I was thinking maybe the documentation doesn't cover it well, page doesn't work, @nasuoa is right, I assume this is intended but I am not sure why, can't believe GitHub would miss it. 🤷🏼

I guess you'll have to settle with this for now, it is not that difficult to implement the pagination with cursor and this is an ideal case for Load More. I am not sure what is your use-case, perhaps you can use events API, it accepts page and per_page and it might solve the issue for you ser.

Good luck. 🍺

🍀

4 replies

nasuoa Nov 6, 2023
Author

Thanks for confirming it's not working using page pagination. The reason I wanted the activities API is to get things like forced pushes, and pushes to specific branches, etc. For my use-case, I was able to deal with it by using the activity_type parameter, by getting specific to which activities I wanted, and luckily those activities had less than 100 results so getting the first page was sufficient.

I'll need to familiarise myself with the cursor navigation, should I need more than 100 results of a specific type.

b8zeek Nov 6, 2023

No worries buddy. ☺️ I am glad you found the solution.

Regarding cursor based pagination, here's an example of the headers I got when I was testing that page thing you were speaking about:

{
  'access-control-allow-origin': '*',
  'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset',
  'cache-control': 'private, max-age=60, s-maxage=60',
  'content-encoding': 'gzip',
  'content-security-policy': "default-src 'none'",
  'content-type': 'application/json; charset=utf-8',
  date: 'Mon, 06 Nov 2023 05:59:32 GMT',
  etag: 'W/"a048cc39e1d9cea1023fed1834ef5cgaf21b77ee32b392bcd9dee11acc2500e3"',
  link: '<https://api.github.com/repositories/189621607/activity?per_page=1&page=10&after=djE6ks8AAAADp2rKWQA>; rel="next"',
  'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
  server: 'GitHub.com',
  'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
  'transfer-encoding': 'chunked',
  vary: 'Accept, Authorization, Cookie, X-GitHub-OTP, Accept-Encoding, Accept, X-Requested-With',
  'x-accepted-oauth-scopes': '',
  'x-content-type-options': 'nosniff',
  'x-frame-options': 'deny',
  'x-github-api-version-selected': '2022-11-28',
  'x-github-media-type': 'github.v3; format=json',
  'x-github-request-id': 'C707:AC90:A1IFAEC:A422842:651910C4',
  'x-oauth-scopes': 'admin:enterprise, admin:gpg_key, admin:org, admin:org_hook, admin:public_key, admin:repo_hook, admin:ssh_signing_key, audit_log, codespace, copilot, delete:packages, delete_repo, gist, notifications, project, repo, user, workflow, write:discussion, write:packages',
  'x-ratelimit-limit': '5000',
  'x-ratelimit-remaining': '0972',
  'x-ratelimit-reset': '1676552544',
  'x-ratelimit-resource': 'core',
  'x-ratelimit-used': '28',
  'x-xss-protection': '0'
}

Here's the Link header:
link: '<https://api.github.com/repositories/189621607/activity?per_page=1&page=10&after=djE6ks8AAAADp2rKWQA>; rel="next"'

If you take a look at it, cursor value is this ➡️ after=djE6ks8AAAADp2rKWQA and you would request the data after that cursor. Link header can contain rel="next", rel="prev", rel="first" and rel="last" and all of those refer to the potentially needed pages. Each of those, if exists points to the link which is stored in your Link header. So, in your case, you would basically check the headers of the response, in case you have rel="next", extract the link value, present the Load More button and fetch the data from that link in case more data is requested.

Hope this clarifies it a bit for you bro. In case you need help with the implementation, reach out or tag me somewhere. ☺️

Take care. 🍺

🍀

Answer selected by nasuoa

nasuoa Nov 7, 2023
Author

Thank you so much! I've learnt something new! This will be extremely helpful. I will aim to have another play with this in a few weeks. Really appreciated! 😄

b8zeek Nov 7, 2023

You're welcome bro.

🍀

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GitHub Community

REST API for "List repository activities" not paginating #73014

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 2 comments 5 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

GitHub Community

REST API for "List repository activities" not paginating #73014

Uh oh!

Uh oh!

nasuoa Oct 31, 2023

GitHub REST API

Replies: 2 comments · 5 replies

Uh oh!

ghostinhershell Nov 6, 2023 Maintainer

Uh oh!

Uh oh!

nasuoa Nov 6, 2023 Author

Uh oh!

b8zeek Nov 6, 2023

Uh oh!

nasuoa Nov 6, 2023 Author

Uh oh!

b8zeek Nov 6, 2023

Uh oh!

nasuoa Nov 7, 2023 Author

Uh oh!

b8zeek Nov 7, 2023

nasuoa
Oct 31, 2023

Replies: 2 comments 5 replies

ghostinhershell
Nov 6, 2023
Maintainer

nasuoa Nov 6, 2023
Author

b8zeek
Nov 6, 2023

nasuoa Nov 6, 2023
Author

nasuoa Nov 7, 2023
Author