Skip to content

Conversation

zeripath
Copy link
Contributor

@zeripath zeripath commented May 19, 2021

Add functionality to pause and resume queues with the ability to hand back unhandled data.

This also now alerts if a queue configuration will result in no workers and no worker scalling (as in #18189) - and will allow no-worker configurations on redis to not result in loss from the queue.

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label May 19, 2021
@zeripath zeripath added this to the 1.15.0 milestone May 19, 2021
@zeripath
Copy link
Contributor Author

FAO @lafriks

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label May 19, 2021
zeripath added 5 commits May 19, 2021 22:03
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
…eues

Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
zeripath and others added 2 commits May 20, 2021 21:32
@lafriks
Copy link
Member

lafriks commented May 20, 2021

From code point of view looks good to me, will test it on weekend together with my WIP PR

@codecov-commenter

This comment has been minimized.

@zeripath zeripath modified the milestones: 1.15.0, 1.16.0 Jun 23, 2021
@zeripath zeripath mentioned this pull request Jul 14, 2021
@lafriks
Copy link
Member

lafriks commented Jul 23, 2021

Build seems to be failing now

@zeripath
Copy link
Contributor Author

yeah I must have made a broken merge when I fixed the last conflict.

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath
Copy link
Contributor Author

fixed

zeripath and others added 3 commits July 23, 2021 16:55
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
Signed-off-by: Andrew Thornton <art27@cantab.net>
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Jan 21, 2022
@zeripath
Copy link
Contributor Author

As an additional thing we can now add warnings for no workers and finally safely handle the case of no worker configurations for redis.

…redis/levelqueue

Signed-off-by: Andrew Thornton <art27@cantab.net>
@zeripath zeripath mentioned this pull request Jan 21, 2022
Copy link
Contributor

@wxiaoguang wxiaoguang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, while maybe util.StopTimer(timer) can be used to reduce some code.

@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Jan 22, 2022
@lafriks
Copy link
Member

lafriks commented Jan 22, 2022

🚀

@zeripath
Copy link
Contributor Author

Make lgtm work

@zeripath zeripath merged commit a82fd98 into go-gitea:main Jan 22, 2022
@zeripath zeripath deleted the pause-queues branch January 22, 2022 22:45
zjjhot pushed a commit to zjjhot/gitea that referenced this pull request Jan 23, 2022
* 'main' of https://github.com/go-gitea/gitea:
  Add packagist webhook (go-gitea#18224)
  Fix mime-type detection for HTTP server (go-gitea#18370)
  Always use git command but not os.Command (go-gitea#18363)
  Add deprecated for LFS_CONTENT_PATH on zh-cn docs (go-gitea#18362)
  [skip ci] Updated translations via Crowdin
  Make gitea, gitea-vet future-proof (go-gitea#18361)
  Pause queues (go-gitea#15928)
  Disable content sniffing on `PlainTextBytes` (go-gitea#18359)
  Update github.com/duo-labs/webauthn (go-gitea#18357)
zeripath added a commit that referenced this pull request Jan 24, 2022
Implements the Pausable interface on WrappedQueues and PersistableChannelUniqueQueues

Reference #15928

Signed-off-by: Andrew Thornton art27@cantab.net
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
* Start adding mechanism to return unhandled data

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Create pushback interface

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add Pausable interface to WorkerPool and Manager

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Implement Pausable and PushBack for the bytefifos

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Implement Pausable and Pushback for ChannelQueues and ChannelUniqueQueues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Wire in UI for pausing

Signed-off-by: Andrew Thornton <art27@cantab.net>

* add testcases and fix a few issues

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix build

Signed-off-by: Andrew Thornton <art27@cantab.net>

* prevent "race" in the test

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix jsoniter mismerge

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix conflicts

Signed-off-by: Andrew Thornton <art27@cantab.net>

* fix format

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Add warnings for no worker configurations and prevent data-loss with redis/levelqueue

Signed-off-by: Andrew Thornton <art27@cantab.net>

* Use StopTimer

Signed-off-by: Andrew Thornton <art27@cantab.net>

Co-authored-by: Lauris BH <lauris@nix.lv>
Co-authored-by: 6543 <6543@obermui.de>
Co-authored-by: techknowlogick <techknowlogick@gitea.io>
Co-authored-by: wxiaoguang <wxiaoguang@gmail.com>
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
…tea#18393)

Implements the Pausable interface on WrappedQueues and PersistableChannelUniqueQueues

Reference go-gitea#15928

Signed-off-by: Andrew Thornton art27@cantab.net
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/feature Completely new functionality. Can only be merged if feature freeze is not active.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants