Improve dashboard's repo list performance #18963

Gusted · 2022-03-01T22:55:37Z

Avoid "a few" database lookups for each repo that's being requested, only return minimal information which doesn't require a database operation for each repo. This is general change for the repo API, so other use-cases of this will also benefit from it.
Makes fetching these list faster.
Less CPU overhead when a user visits home page.
Unintentionally removed some jQuery code.

- Avoid a lot of database lookups for all the repo's, by adding a undocumented "minimal" mode for this specific task, which returns the data that's only needed by this list which doesn't require any database lookups. - Makes fetching these list faster. - Less CPU overhead when a user visits home page.

web_src/js/components/DashboardRepoList.js

lunny · 2022-03-02T00:41:06Z

I would like we split api/v1 and UI API to make the things simple. This is a chance to make a new route but not reuse the old one.

Gusted · 2022-03-02T10:41:43Z

I would like we split api/v1 and UI API to make the things simple. This is a chance to make a new route but not reuse the old one.

Sounds like a good idea to me, which route do you propose for it? I guess api/internal/... or api/ui/...?

- Use async in the function so we can use `await`. - Remove `archivedFilter` check for count, as it doesn't make sense to show the count of repos when you can't even see them(as they are filited away).

web_src/js/components/DashboardRepoList.js

6543 · 2022-03-03T16:14:15Z

... -> #16052

6543 · 2022-03-14T11:08:33Z

I'm for adding a new ui route - we can migrate one func at a time ... just start now

(UI routes dont have the issue to need to be/stay backwards compatible)

Gusted · 2022-03-14T13:03:27Z

just start now

I don't think that's possible, #16052 adds many new things and as well it's own file for the new API, it will result in conflict files for that PR if I try remotely to migrate it to a new route.

6543 · 2022-04-07T19:01:14Z

#19318 got merged ... please ajust this pull @Gusted

Gusted · 2022-04-11T22:20:50Z

@6543 Updated it.

routers/api/v1/repo/repo.go

codecov-commenter · 2022-04-12T15:58:57Z

Codecov Report

Merging #18963 (7319f8d) into main (deffe9e) will increase coverage by 0.00%.
The diff coverage is 0.00%.

@@           Coverage Diff           @@
##             main   #18963   +/-   ##
=======================================
  Coverage   47.37%   47.37%           
=======================================
  Files         949      949           
  Lines      132101   132102    +1     
=======================================
+ Hits        62583    62588    +5     
+ Misses      61987    61982    -5     
- Partials     7531     7532    +1

Impacted Files	Coverage Δ
routers/api/v1/repo/repo.go	`66.55% <ø> (ø)`
routers/web/repo/repo.go	`24.37% <0.00%> (-0.06%)`	⬇️
modules/process/manager_exec.go	`85.36% <0.00%> (-7.32%)`	⬇️
models/unit/unit.go	`46.08% <0.00%> (-1.74%)`	⬇️
models/repo_list.go	`76.15% <0.00%> (+0.48%)`	⬆️
modules/queue/workerpool.go	`53.62% <0.00%> (+2.07%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update deffe9e...7319f8d. Read the comment docs.

routers/web/repo/repo.go

web_src/js/components/DashboardRepoList.js

Co-authored-by: delvh <dev.lh@web.de>

routers/web/repo/repo.go

Co-authored-by: delvh <dev.lh@web.de>

routers/web/repo/repo.go

web_src/js/components/DashboardRepoList.js

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

wxiaoguang

CI seems down .... wait for the pass

zeripath · 2022-04-26T17:54:19Z

make lgtm work

* giteaofficial/main: (21 commits) Prevent intermittent race in attribute reader close (go-gitea#19537) Make repository file list useable on mobile (go-gitea#19515) Update image URL for Discord webhook (go-gitea#19536) [skip ci] Updated translations via Crowdin Fix 64-bit atomic operations on 32-bit machines (go-gitea#19531) Fix `upgrade.sh` script error with `su -c` (go-gitea#19483) When view _Siderbar or _Footer, just display once (go-gitea#19501) Fix migrate release from github (go-gitea#19510) Prevent dangling archiver goroutine (go-gitea#19516) Don't let repo clone URL overflow (go-gitea#19517) Add commit status popup to issuelist (go-gitea#19375) Disable unnecessary GitHooks elements (go-gitea#18485) Disable unnecessary GitHooks elements Improve dashboard's repo list performance (go-gitea#18963) By default force vertical tabs on mobile (go-gitea#19486) Refactor readme file renderer (go-gitea#19502) Allow package dump skipping (go-gitea#19506) Unset git author/committer variables when running integration tests (go-gitea#19512) Allow commit status popup on /pulls page (go-gitea#19507) Use router param for filepath in GetRawFile (go-gitea#19499) ...

* Improve dashboard's repo list performance - Avoid a lot of database lookups for all the repo's, by adding a undocumented "minimal" mode for this specific task, which returns the data that's only needed by this list which doesn't require any database lookups. - Makes fetching these list faster. - Less CPU overhead when a user visits home page. * Refactor javascript code + fix Fork icon - Use async in the function so we can use `await`. - Remove `archivedFilter` check for count, as it doesn't make sense to show the count of repos when you can't even see them(as they are filited away). * Add `count_only` * Remove uncessary code * Improve comment Co-authored-by: delvh <dev.lh@web.de> * Update web_src/js/components/DashboardRepoList.js Co-authored-by: delvh <dev.lh@web.de> * Update web_src/js/components/DashboardRepoList.js Co-authored-by: delvh <dev.lh@web.de> * By default apply minimal mode * Remove `minimal` paramater * Refactor count header * Simplify init Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: delvh <dev.lh@web.de> Co-authored-by: wxiaoguang <wxiaoguang@gmail.com> Co-authored-by: zeripath <art27@cantab.net>

Gusted added this to the 1.17.0 milestone Mar 1, 2022

Gusted added the performance/cpu label Mar 1, 2022

Gusted commented Mar 1, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Mar 1, 2022

Gusted added 2 commits March 2, 2022 12:10

Refactor javascript code + fix Fork icon

22acfd6

- Use async in the function so we can use `await`. - Remove `archivedFilter` check for count, as it doesn't make sense to show the count of repos when you can't even see them(as they are filited away).

Merge branch 'main' into improve-dashboard-list

931de98

wxiaoguang reviewed Mar 2, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

wxiaoguang reviewed Mar 2, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

Add count_only

c6706e6

silverwind approved these changes Mar 2, 2022

View reviewed changes

GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Mar 2, 2022

Merge branch 'main' into improve-dashboard-list

964527b

lafriks reviewed Apr 12, 2022

View reviewed changes

routers/api/v1/repo/repo.go Outdated Show resolved Hide resolved

Remove uncessary code

53199d4

delvh reviewed Apr 12, 2022

View reviewed changes

Gusted and others added 2 commits April 14, 2022 16:37

Improve comment

47926f5

Co-authored-by: delvh <dev.lh@web.de>

Update web_src/js/components/DashboardRepoList.js

1b42dd2

Co-authored-by: delvh <dev.lh@web.de>

wxiaoguang reviewed Apr 15, 2022

View reviewed changes

routers/web/repo/repo.go Outdated Show resolved Hide resolved

Gusted and others added 2 commits April 15, 2022 10:47

Update web_src/js/components/DashboardRepoList.js

1444f06

Co-authored-by: delvh <dev.lh@web.de>

By default apply minimal mode

19f811f

Gusted added 2 commits April 15, 2022 19:28

Remove minimal paramater

3b67a8c

Merge branch 'main' into improve-dashboard-list

7319f8d

delvh reviewed Apr 15, 2022

View reviewed changes

routers/web/repo/repo.go Outdated Show resolved Hide resolved

web_src/js/components/DashboardRepoList.js Show resolved Hide resolved

Gusted added 2 commits April 25, 2022 21:35

Refactor count header

3b91ad4

Merge branch 'main' into improve-dashboard-list

84bf230

delvh approved these changes Apr 26, 2022

View reviewed changes

GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Apr 26, 2022

Merge branch 'main' into improve-dashboard-list

cb4f7d2

wxiaoguang reviewed Apr 26, 2022

View reviewed changes

web_src/js/components/DashboardRepoList.js Outdated Show resolved Hide resolved

Simplify init

fab1d89

Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>

wxiaoguang approved these changes Apr 26, 2022

View reviewed changes

Merge branch 'main' into improve-dashboard-list

7594040

Merge branch 'main' into improve-dashboard-list

b0ded7f

techknowlogick merged commit 076eaad into go-gitea:main Apr 26, 2022

Gusted deleted the improve-dashboard-list branch April 26, 2022 20:35

go-gitea locked and limited conversation to collaborators May 3, 2023

Uh oh!

Improve dashboard's repo list performance #18963

Improve dashboard's repo list performance #18963

Uh oh!

Conversation

Gusted commented Mar 1, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

lunny commented Mar 2, 2022

Uh oh!

Gusted commented Mar 2, 2022

Uh oh!

Uh oh!

Uh oh!

6543 commented Mar 3, 2022

Uh oh!

6543 commented Mar 14, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Gusted commented Mar 14, 2022

Uh oh!

6543 commented Apr 7, 2022

Uh oh!

Gusted commented Apr 11, 2022

Uh oh!

Uh oh!

codecov-commenter commented Apr 12, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wxiaoguang left a comment

Choose a reason for hiding this comment

Uh oh!

zeripath commented Apr 26, 2022

Uh oh!

Uh oh!

Gusted commented Mar 1, 2022 •

edited

Loading

6543 commented Mar 14, 2022 •

edited

Loading

codecov-commenter commented Apr 12, 2022 •

edited

Loading