Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: huggingface/xet-core
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.1.4
Choose a base ref
...
head repository: huggingface/xet-core
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.1.5
Choose a head ref
  • 5 commits
  • 14 files changed
  • 3 contributors

Commits on Jun 17, 2025

  1. Shard interface updates (#382)

    Changes to be used potentially in a CAS server PR.
    
    - consistent usage of futures::io::AsyncRead and import pattern
    - add deserialize_async variants to cas info and file info used structs.
    The only difference is the use of async readers, but we still just read
    the whole struct worth (expect top level) of data and deserialize from
    slice.
    - constants exports
    assafvayner authored Jun 17, 2025
    Configuration menu
    Copy the full SHA
    8c2bbaa View commit details
    Browse the repository at this point in the history
  2. Background loading for shards (#384)

    Currently, loading all of the shards is done in a blocking manner, which
    means that a large number of shards causes the call to upload_files to
    take a long time to get started. This PR optimizes this path by loading
    the lookup table sections of the shard directories in the background
    while the chunking and file reading can get started.
    
    It also introduces a new utility class, RwTaskLock, which provides a
    RwLock-like interface around a value that can either be specified by the
    value itself or by a future that resolves to the value. This makes it
    easy to background tasks when values like lookup tables are held behind
    an rwlock-like interface. This utility is self-contained and unit tests
    are provided.
    hoytak authored Jun 17, 2025
    Configuration menu
    Copy the full SHA
    7f89855 View commit details
    Browse the repository at this point in the history
  3. fix MDBFileInfo::deserialize_async in case of no verification entries (

    …#388)
    
    fixes issue in #382, where if a file info has not verification info then
    deserialization would be be incorrect.
    assafvayner authored Jun 17, 2025
    Configuration menu
    Copy the full SHA
    7d6301f View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2025

  1. Configuration menu
    Copy the full SHA
    2cdc186 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d55c6a2 View commit details
    Browse the repository at this point in the history
Loading