Skip to content

compaction failed, suspected data corruption issue #16490

New issue
New issue
Closed
#16623
Closed
compaction failed, suspected data corruption issue#16490
#16623
Labels
component/scrapingcomponent/tsdbkind/bug
@ipetkov

Description

@ipetkov
ipetkov
opened on Apr 27, 2025

What did you do?

I upgraded Prometheus from 3.1.0 to 3.3.0 (running on NixOS) and a few hours later I noticed a bunch of my metrics had disappeared and the logs were full of errors like:

Apr 26 19:05:56 elysium prometheus[229018]: time=2025-04-27T02:05:56.022Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"

I stopped Prometheus and deleted its data and restarted it. At first everything hummed along but a few hours later the same issue came up. There was no power loss/forceful shutdown that could have resulted in the files being corrupted so I suspect there might be a different issue lurking. I am using OpenZFS as my filesystem:

zfs-2.3.1-1
zfs-kmod-2.3.1-1

What did you expect to see?

No corruption/missing metrics

What did you see instead? Under which circumstances?

Apr 26 19:05:56 elysium prometheus[229018]: time=2025-04-27T02:05:56.022Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"

This seems to happen a few hours after clearing all data and starting Prometheus. I suspect it might be tied to WAL compaction/GC events since it does not trigger immediately

System information

Linux 6.12.24 x86_64

Prometheus version

prometheus, version 3.3.0 (branch: unknown, revision: unknown)
  build user:       nix@nixpkgs
  build date:       unknown
  go version:       go1.24.2
  platform:         linux/amd64
  tags:             builtinassets

Prometheus configuration file

alerting:
  alertmanagers: []
global:
  scrape_interval: 1m
remote_read: []
remote_write: []
rule_files:
# This file happens to be blank
- /nix/store/k9yji2pyc8swnjlxxpgn5vaizf495hhg-rules-checkrules-checked
scrape_configs:
- job_name: prod
  static_configs:
  - labels: {}
    targets:
    - asphodel:9100
    - asphodel:9134
    - asphodel:9633
    - asphodel:9999
    - elysium:9999
    - elysium:9100
    - elysium:9633
    - elysium:9134
    - rpi:9100
    - rpi:9119
    - rpi:9999
- job_name: dev
  static_configs:
  - labels: {}
    targets:
    - tartarus:9100
    - tartarus:9134
    - tartarus:9633

Alertmanager version

Alertmanager configuration file



Logs


Apr 26 16:05:38 elysium systemd[1]: Started prometheus.service.
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.167Z level=INFO source=main.go:687 msg="Starting Prometheus Server" mode=server version="(version=3.3.0, branch=unknown, revision=unknown)"
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.167Z level=INFO source=main.go:692 msg="operational information" build_context="(go=go1.24.2, platform=linux/amd64, user=nix@nixpkgs, date=unknown, tags=builtinassets)" host_details="(Linux 6.12.24 #1-NixOS SMP PREEMPT_DYNAMIC Sun Apr 20 08:16:02 UTC 2025 x86_64 elysium (none))" fd_limits="(soft=524287, hard=524288)" vm_limits="(soft=unlimited, hard=unlimited)"
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.168Z level=INFO source=main.go:768 msg="Leaving GOMAXPROCS=16: CPU quota undefined" component=automaxprocs
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.169Z level=INFO source=web.go:654 msg="Start listening for connections" component=web address=0.0.0.0:9090
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.169Z level=INFO source=main.go:1228 msg="Starting TSDB ..."
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.169Z level=INFO source=tls_config.go:347 msg="Listening on" component=web address=[::]:9090
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.169Z level=INFO source=tls_config.go:350 msg="TLS is disabled." component=web http2=false address=[::]:9090
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.171Z level=INFO source=head.go:638 msg="Replaying on-disk memory mappable chunks if any" component=tsdb
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.171Z level=INFO source=head.go:725 msg="On-disk memory mappable chunks replay completed" component=tsdb duration=702ns
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.171Z level=INFO source=head.go:733 msg="Replaying WAL, this may take a while" component=tsdb
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.171Z level=INFO source=head.go:805 msg="WAL segment loaded" component=tsdb segment=0 maxSegment=0
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.171Z level=INFO source=head.go:842 msg="WAL replay completed" component=tsdb checkpoint_replay_duration=19.326µs wal_replay_duration=333.663µs wbl_replay_duration=150ns chunk_snapshot_load_duration=0s mmap_chunk_replay_duration=702ns total_replay_duration=365.442µs
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.172Z level=INFO source=main.go:1249 msg="filesystem information" fs_type=2fc12fc1
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.172Z level=INFO source=main.go:1252 msg="TSDB started"
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.172Z level=INFO source=main.go:1437 msg="Loading configuration file" filename=/nix/store/3cnm0xkjvzzi9r2946gjqlf80gkzg2qi-prometheus.yml-checkconfig-checked
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.173Z level=INFO source=main.go:1476 msg="updated GOGC" old=100 new=75
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.173Z level=INFO source=main.go:1486 msg="Completed loading of configuration file" db_storage=501ns remote_storage=561ns web_handler=90ns query_engine=561ns scrape=357.226µs scrape_sd=16.531µs notify=451ns notify_sd=310ns rules=32.731µs tracing=2.615µs filename=/nix/store/3cnm0xkjvzzi9r2946gjqlf80gkzg2qi-prometheus.yml-checkconfig-checked totalDuration=538.876µs
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.173Z level=INFO source=main.go:1213 msg="Server is ready to receive web requests."
Apr 26 16:05:38 elysium prometheus[229018]: time=2025-04-26T23:05:38.173Z level=INFO source=manager.go:175 msg="Starting rule manager..." component="rule manager"
Apr 26 19:05:56 elysium prometheus[229018]: time=2025-04-27T02:05:56.022Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:05:58 elysium prometheus[229018]: time=2025-04-27T02:05:58.543Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:06:07 elysium prometheus[229018]: time=2025-04-27T02:06:07.484Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:06:11 elysium prometheus[229018]: time=2025-04-27T02:06:11.506Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 49855"
Apr 26 19:06:19 elysium prometheus[229018]: time=2025-04-27T02:06:19.526Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 49839"
Apr 26 19:06:35 elysium prometheus[229018]: time=2025-04-27T02:06:35.557Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 49839"
Apr 26 19:07:07 elysium prometheus[229018]: time=2025-04-27T02:07:07.577Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:08:07 elysium prometheus[229018]: time=2025-04-27T02:08:07.597Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:09:07 elysium prometheus[229018]: time=2025-04-27T02:09:07.621Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 49839"
Apr 26 19:10:07 elysium prometheus[229018]: time=2025-04-27T02:10:07.644Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:11:07 elysium prometheus[229018]: time=2025-04-27T02:11:07.663Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:12:07 elysium prometheus[229018]: time=2025-04-27T02:12:07.687Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 51919"
Apr 26 19:13:07 elysium prometheus[229018]: time=2025-04-27T02:13:07.710Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:14:07 elysium prometheus[229018]: time=2025-04-27T02:14:07.727Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:15:07 elysium prometheus[229018]: time=2025-04-27T02:15:07.750Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:16:07 elysium prometheus[229018]: time=2025-04-27T02:16:07.770Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:17:07 elysium prometheus[229018]: time=2025-04-27T02:17:07.791Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:18:07 elysium prometheus[229018]: time=2025-04-27T02:18:07.811Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:19:07 elysium prometheus[229018]: time=2025-04-27T02:19:07.824Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:20:07 elysium prometheus[229018]: time=2025-04-27T02:20:07.844Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:21:07 elysium prometheus[229018]: time=2025-04-27T02:21:07.863Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:22:07 elysium prometheus[229018]: time=2025-04-27T02:22:07.882Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:23:07 elysium prometheus[229018]: time=2025-04-27T02:23:07.903Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:24:07 elysium prometheus[229018]: time=2025-04-27T02:24:07.930Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:25:07 elysium prometheus[229018]: time=2025-04-27T02:25:07.954Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:26:07 elysium prometheus[229018]: time=2025-04-27T02:26:07.975Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:27:07 elysium prometheus[229018]: time=2025-04-27T02:27:07.995Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:28:08 elysium prometheus[229018]: time=2025-04-27T02:28:08.015Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:29:08 elysium prometheus[229018]: time=2025-04-27T02:29:08.036Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:30:08 elysium prometheus[229018]: time=2025-04-27T02:30:08.055Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:31:08 elysium prometheus[229018]: time=2025-04-27T02:31:08.075Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:32:08 elysium prometheus[229018]: time=2025-04-27T02:32:08.093Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:33:08 elysium prometheus[229018]: time=2025-04-27T02:33:08.117Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:34:08 elysium prometheus[229018]: time=2025-04-27T02:34:08.143Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:35:08 elysium prometheus[229018]: time=2025-04-27T02:35:08.168Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:36:08 elysium prometheus[229018]: time=2025-04-27T02:36:08.186Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"tsnet\" does not exist; series not 16-byte aligned at 52911"
Apr 26 19:37:08 elysium prometheus[229018]: time=2025-04-27T02:37:08.207Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:38:08 elysium prometheus[229018]: time=2025-04-27T02:38:08.224Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"NXDOMAI\" does not exist; series not 16-byte aligned at 50831"
Apr 26 19:39:08 elysium prometheus[229018]: time=2025-04-27T02:39:08.275Z level=INFO source=compact.go:591 msg="write block" component=tsdb mint=1745708754554 maxt=1745712000000 ulid=01JSTHSQE1NYKRFJNXJ4T09GJQ duration=50.512582ms ooo=false
Apr 26 19:39:08 elysium prometheus[229018]: time=2025-04-27T02:39:08.278Z level=INFO source=head.go:1400 msg="Head GC completed" component=tsdb caller=truncateMemory duration=2.439858ms
Apr 26 20:00:11 elysium prometheus[229018]: time=2025-04-27T03:00:11.565Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:00:12 elysium prometheus[229018]: time=2025-04-27T03:00:12.868Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:00:14 elysium prometheus[229018]: time=2025-04-27T03:00:14.894Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:00:18 elysium prometheus[229018]: time=2025-04-27T03:00:18.917Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50927"
Apr 26 20:00:26 elysium prometheus[229018]: time=2025-04-27T03:00:26.941Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:00:42 elysium prometheus[229018]: time=2025-04-27T03:00:42.969Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:01:14 elysium prometheus[229018]: time=2025-04-27T03:01:14.992Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50911"
Apr 26 20:02:15 elysium prometheus[229018]: time=2025-04-27T03:02:15.015Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:03:15 elysium prometheus[229018]: time=2025-04-27T03:03:15.036Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50911"
Apr 26 20:04:15 elysium prometheus[229018]: time=2025-04-27T03:04:15.055Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50927"
Apr 26 20:05:15 elysium prometheus[229018]: time=2025-04-27T03:05:15.076Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50911"
Apr 26 20:06:15 elysium prometheus[229018]: time=2025-04-27T03:06:15.096Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50911"
Apr 26 20:07:15 elysium prometheus[229018]: time=2025-04-27T03:07:15.122Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:08:15 elysium prometheus[229018]: time=2025-04-27T03:08:15.148Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50927"
Apr 26 20:09:15 elysium prometheus[229018]: time=2025-04-27T03:09:15.172Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:10:15 elysium prometheus[229018]: time=2025-04-27T03:10:15.193Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50927"
Apr 26 20:11:15 elysium prometheus[229018]: time=2025-04-27T03:11:15.212Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50927"
Apr 26 20:12:15 elysium prometheus[229018]: time=2025-04-27T03:12:15.238Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 50927"
Apr 26 20:13:15 elysium prometheus[229018]: time=2025-04-27T03:13:15.264Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="add series: out-of-order series added with label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\", last label set \"{__name__=\\\"\\\\nnode_filesystem_device_erro\\\", \\\"{devic\\\"=\\\"\\\\\\\"tmpfs\\\", fstype=\\\"tmpf\\\", instance=\\\"tartarus:9100\\\", job=\\\"dev\\\", \\\",mountpoin\\\"=\\\"\\\\\\\"/ru\\\"}\""
Apr 26 20:14:15 elysium prometheus[229018]: time=2025-04-27T03:14:15.287Z level=ERROR source=db.go:1114 msg="compaction failed" component=tsdb err="2 errors: add series: symbol entry for \"\\\"!SVC\" does not exist; series not 16-byte aligned at 51855"

Metadata
Metadata
Assignees
No one assigned
    Labels
    component/scrapingcomponent/tsdbkind/bug
    Type
    No type
    Projects
    No projects
    Milestone
    No milestone
    Relationships
    None yet
    Development
    No branches or pull requests
    Issue actions