Skip to content

Conversation

machine424
Copy link
Member

@machine424 machine424 commented Nov 8, 2024

See proposal prometheus/proposals#45

  • Create needed issues

@machine424
Copy link
Member Author

/prombench main

@prombot
Copy link
Contributor

prombot commented Nov 8, 2024

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-15365 and main

After successful deployment, the benchmarking results can be viewed at:

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart main

@machine424
Copy link
Member Author

/prombench cancel

@prombot
Copy link
Contributor

prombot commented Nov 8, 2024

Benchmark cancel is in progress.

@machine424 machine424 force-pushed the dio branch 3 times, most recently from 125e4a9 to 3c6fec4 Compare December 4, 2024 13:46
@machine424
Copy link
Member Author

/prombench main

@prombot
Copy link
Contributor

prombot commented Dec 4, 2024

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-15365 and main

After successful deployment, the benchmarking results can be viewed at:

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart main

@machine424
Copy link
Member Author

/prombench cancel

@prombot
Copy link
Contributor

prombot commented Dec 4, 2024

Benchmark cancel is in progress.

@machine424
Copy link
Member Author

/prombench main

@prombot
Copy link
Contributor

prombot commented Dec 4, 2024

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-15365 and main

After successful deployment, the benchmarking results can be viewed at:

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart main

@machine424
Copy link
Member Author

/prombench cancel

@prombot
Copy link
Contributor

prombot commented Dec 5, 2024

Benchmark cancel is in progress.

@machine424
Copy link
Member Author

/prombench main

@prombot
Copy link
Contributor

prombot commented Dec 5, 2024

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-15365 and main

After successful deployment, the benchmarking results can be viewed at:

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart main

@prombot
Copy link
Contributor

prombot commented Dec 8, 2024

Benchmark tests are running for 3 days! If this is intended ignore this message otherwise you can cancel it by commenting: /prombench cancel

@prombot
Copy link
Contributor

prombot commented Dec 10, 2024

Benchmark tests are running for 5 days! If this is intended ignore this message otherwise you can cancel it by commenting: /prombench cancel

@bboreham
Copy link
Member

bboreham commented Jan 2, 2025

Might be useful for other operations (WAL writing?

Would this require holding WAL records in memory until there is 4KB to write?
That gives me mixed feelings.

@machine424
Copy link
Member Author

machine424 commented Jan 3, 2025

Might be useful for other operations (WAL writing?

Would this require holding WAL records in memory until there is 4KB to write? That gives me mixed feelings.

Yes, theoretically we could omit writing up to "offsetAlign-1" of data, we could work around that, but it might result in unwanted side effects like write amplification, bloat, and switching between direct and buffered I/O...

That being said, we already write in blocks of pageSize=32KB, so the alignment should be ensured in most cases.

I'll need to look into this in more detail to see if it's really worth it.

@machine424 machine424 marked this pull request as ready for review February 18, 2025 12:24
@machine424 machine424 changed the title feat(tsdb): allow using Direct IO for chuncks segments writing feat(tsdb): allow using Direct IO for chunks writing Feb 18, 2025
@bboreham
Copy link
Member

Adding the metric container_memory_usage_bytes, which a lot of people look at, to the Prombench dashboard we see that it is significantly lower with Direct IO.

image

@seizethedave
Copy link

Adding the metric container_memory_usage_bytes, which a lot of people look at

Cool. What happens if you plot just file cache memory?

Another thing that DIO can buy is placing less pressure on page cache evictions and thus reducing the number of major page faults occurring in the places using mapped IO. Could be another useful metric to look at.

@machine424
Copy link
Member Author

machine424 commented Apr 18, 2025

What happens if you plot just file cache memory?

There is one in here https://github.com/prometheus/proposals/pull/45/files
Yes, less page cache should/will imply less page cache eviction.

@machine424 machine424 force-pushed the dio branch 3 times, most recently from ae808d1 to 60dfde2 Compare April 23, 2025 09:35
Copy link
Member

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a lot of deep magic here; I did not attempt to follow all of it.

Signed-off-by: machine424 <ayoubmrini424@gmail.com>
@machine424
Copy link
Member Author

force push + rebase, the diff wouldn't be helpful :(

@machine424
Copy link
Member Author

without checking the conflicts, I'd say that a new feature flag was added, again.
I'll resolve them, hope it's the last time.

Signed-off-by: Ayoub Mrini <ayoubmrini424@gmail.com>
Copy link
Member

@bboreham bboreham left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@machine424 machine424 merged commit 2edc3ed into prometheus:main May 21, 2025
27 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants