Skip to content

Conversation

ptpt
Copy link
Member

@ptpt ptpt commented Sep 4, 2025

  • Added a new module store.py based on dbm.sqlite3(python3.13 only), available for python3.9+,
  • which means all file handles are cached in the temporary sqlite3 DB with locks when writing
  • Lots of tests added to ensure it works cross platform 🙏

There are some delays in cache operations like get and set (especially set, ~30ms), which may affect bandwidth utilization for single-threaded uploading (i.e. --num_upload_workers=1). For multiple workers uploading, it should be fine.

@meta-cla meta-cla bot added the cla signed label Sep 4, 2025
@ptpt ptpt merged commit 25a08de into main Sep 4, 2025
16 checks passed
@ptpt ptpt deleted the fix-database-lock branch September 4, 2025 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant