Skip to content

nixos/loki: Added Loki scalable deployment mode testcase #313189

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 4 commits into
base: master
Choose a base branch
from
Draft

nixos/loki: Added Loki scalable deployment mode testcase #313189

wants to merge 4 commits into from

Conversation

jpds
Copy link
Contributor

@jpds jpds commented May 20, 2024

Description of changes

Adds a Loki scalable deployment mode testcase to the NixOS tests.

Things done

  • Built on platform(s)
    • x86_64-linux
    • aarch64-linux
    • x86_64-darwin
    • aarch64-darwin
  • For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
    • sandbox = relaxed
    • sandbox = true
  • Tested, as applicable:
  • Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
  • Tested basic functionality of all binary files (usually in ./result/bin/)
  • 24.05 Release Notes (or backporting 23.05 and 23.11 Release notes)
    • (Package updates) Added a release notes entry if the change is major or breaking
    • (Module updates) Added a release notes entry if the change is significant
    • (Module addition) Added a release notes entry if adding a new NixOS module
  • Fits CONTRIBUTING.md.

Add a 👍 reaction to pull requests you find important.

@github-actions github-actions bot added the 6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS label May 20, 2024
@cole-h
Copy link
Member

cole-h commented May 20, 2024

@ofborg eval

@jpds jpds force-pushed the nixos-test-loki-ssd branch from 8c9f0ab to 866a1c2 Compare May 20, 2024 23:31
@ofborg ofborg bot added 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. labels May 20, 2024
@jpds jpds force-pushed the nixos-test-loki-ssd branch 2 times, most recently from 8804fd1 to 68c602c Compare May 21, 2024 10:22
@jpds
Copy link
Contributor Author

jpds commented May 21, 2024

@WilliButz @globin @mmahut @emilylange This one should be ready to go.

@jpds jpds force-pushed the nixos-test-loki-ssd branch from 68c602c to 48cdfde Compare May 21, 2024 23:33
@Scrumplex
Copy link
Member

@ofborg test loki.standalone loki.ssd

Scrumplex
Scrumplex approved these changes Oct 6, 2024
View reviewed changes
Copy link
Member

@Scrumplex Scrumplex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Both tests succeed locally.

@wegank wegank added the 12.approvals: 1 This PR was reviewed and approved by one person. label Oct 29, 2024
@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Dec 10, 2024
@jpds jpds force-pushed the nixos-test-loki-ssd branch 2 times, most recently from 85c8e8c to a8f265d Compare December 19, 2024 17:35
@wegank wegank removed the 12.approvals: 1 This PR was reviewed and approved by one person. label Dec 19, 2024
@ofborg ofborg bot removed the 2.status: merge conflict This PR has merge conflicts with the target branch label Dec 20, 2024
@globin
Copy link
Member

globin commented Dec 20, 2024

@ofborg test loki.standalone loki.ssd

@wegank wegank added the 2.status: merge conflict This PR has merge conflicts with the target branch label Apr 2, 2025
@nixpkgs-ci nixpkgs-ci bot added 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md 12.approvals: 1 This PR was reviewed and approved by one person. labels Jun 25, 2025
@jpds jpds force-pushed the nixos-test-loki-ssd branch from a8f265d to 1984d9b Compare August 17, 2025 14:36
@nixpkgs-ci nixpkgs-ci bot removed 2.status: merge conflict This PR has merge conflicts with the target branch 2.status: stale https://github.com/NixOS/nixpkgs/blob/master/.github/STALE-BOT.md labels Aug 17, 2025
@jpds jpds force-pushed the nixos-test-loki-ssd branch from 1984d9b to 83d8048 Compare August 17, 2025 14:44
misuzu
misuzu reviewed Aug 18, 2025
View reviewed changes
nixos/tests/loki/default.nix

{
standalone = runTest ./standalone.nix;
ssd = runTest ./ssd.nix;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test looks broken:

read # [  238.803936] loki[937]: level=error ts=2025-08-18T16:41:15.923725893Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  238.900562] loki[937]: level=error ts=2025-08-18T16:41:16.06238793Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  239.100898] loki[937]: level=error ts=2025-08-18T16:41:16.253756106Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  239.700635] loki[937]: level=error ts=2025-08-18T16:41:16.842703026Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.0.2.15:9095"
read # [  240.301037] loki[937]: level=error ts=2025-08-18T16:41:17.446715826Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  240.401058] loki[937]: level=error ts=2025-08-18T16:41:17.557993935Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  240.601561] loki[937]: level=error ts=2025-08-18T16:41:17.746104715Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  241.300586] loki[937]: level=error ts=2025-08-18T16:41:18.400852455Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  242.601306] loki[937]: level=error ts=2025-08-18T16:41:19.70544843Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
write # [  223.922415] loki[938]: level=info ts=2025-08-18T16:41:20.131051407Z caller=recalculate_owned_streams.go:49 msg="starting recalculate owned streams job"
write # [  223.924704] loki[938]: level=info ts=2025-08-18T16:41:20.131120969Z caller=recalculate_owned_streams.go:52 msg="completed recalculate owned streams job"
read # [  242.701305] loki[937]: level=error ts=2025-08-18T16:41:19.842202682Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.0.2.15:9095"
write # [  224.022897] loki[938]: level=info ts=2025-08-18T16:41:20.206472166Z caller=memberlist_client.go:552 msg="initiating cleanup of obsolete entries"
read # [  243.400339] loki[937]: level=error ts=2025-08-18T16:41:20.517408889Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  243.404323] loki[937]: level=error ts=2025-08-18T16:41:20.533069538Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  243.701517] loki[937]: level=error ts=2025-08-18T16:41:20.864137593Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  243.801255] loki[937]: level=error ts=2025-08-18T16:41:20.961244069Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  244.401131] loki[937]: level=error ts=2025-08-18T16:41:21.504943617Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  244.700678] loki[937]: level=error ts=2025-08-18T16:41:21.830209538Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  245.501776] loki[937]: level=error ts=2025-08-18T16:41:22.663028234Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  245.700803] loki[937]: level=error ts=2025-08-18T16:41:22.841897615Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.0.2.15:9095"
backend # [  271.798989] loki[949]: level=info ts=2025-08-18T16:41:23.804974204Z caller=memberlist_client.go:552 msg="initiating cleanup of obsolete entries"
read # [  246.200326] loki[937]: level=error ts=2025-08-18T16:41:23.352353274Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  247.100640] loki[937]: level=error ts=2025-08-18T16:41:24.20194142Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  247.400887] loki[937]: level=error ts=2025-08-18T16:41:24.555731357Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  247.701132] loki[937]: level=error ts=2025-08-18T16:41:24.84195265Z caller=ring_watcher.go:56 component=frontend-scheduler-worker msg="error getting addresses from ring" err="at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.0.2.15:9095"
read # [  247.900648] loki[937]: level=error ts=2025-08-18T16:41:25.016170044Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  248.000592] loki[937]: level=error ts=2025-08-18T16:41:25.147626416Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  248.400675] loki[937]: level=error ts=2025-08-18T16:41:25.590282116Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  248.701126] loki[937]: level=error ts=2025-08-18T16:41:25.841958237Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.0.2.15:9095"
read # [  249.101335] loki[937]: level=error ts=2025-08-18T16:41:26.257149833Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  249.300997] loki[937]: level=error ts=2025-08-18T16:41:26.419240545Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  249.901090] loki[937]: level=error ts=2025-08-18T16:41:27.027221727Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  250.301205] loki[937]: level=error ts=2025-08-18T16:41:27.497127539Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  251.401534] loki[937]: level=error ts=2025-08-18T16:41:28.546118403Z caller=scheduler_processor.go:111 component=querier msg="error processing requests from scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForQuerier" addr=10.0.2.15:9095
read # [  251.700645] loki[937]: level=error ts=2025-08-18T16:41:28.842602173Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="at least 1 healthy replica required, could only find 0 - unhealthy instances: 10.0.2.15:9095"
read # [  252.100387] loki[937]: level=error ts=2025-08-18T16:41:29.204385864Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  252.103428] loki[937]: level=error ts=2025-08-18T16:41:29.237118795Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
read # [  252.201355] loki[937]: level=error ts=2025-08-18T16:41:29.345321374Z caller=frontend_scheduler_worker.go:249 msg="error sending requests to scheduler" err="rpc error: code = Unimplemented desc = unknown service schedulerpb.SchedulerForFrontend" addr=10.0.2.15:9095
^Cerror: interrupted by the user

@jpds jpds marked this pull request as draft August 19, 2025 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@misuzu misuzu misuzu left review comments

@Scrumplex Scrumplex Scrumplex approved these changes

@mmahut mmahut Awaiting requested review from mmahut

@globin globin Awaiting requested review from globin

@WilliButz WilliButz Awaiting requested review from WilliButz

@emilylange emilylange Awaiting requested review from emilylange

Assignees
No one assigned
Labels
6.topic: nixos Issues or PRs affecting NixOS modules, or package usability issues specific to NixOS 10.rebuild-darwin: 0 This PR does not cause any packages to rebuild on Darwin. 10.rebuild-linux: 0 This PR does not cause any packages to rebuild on Linux. 12.approvals: 1 This PR was reviewed and approved by one person.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@jpds @cole-h @Scrumplex @globin @misuzu @wegank