Fixes #6025

When loading a local shard, only load segments from valid directory paths.

Before, we'd try to load every entry in the segments directory as if it was a segment. This caused a panic during segment loading if macOS places a .DS_Store file in there.

More specifically, we now ignore:

• non-directories
• hidden directories prefixed with a .
• .DS_Store

We now also walk over all segment path entries before trying to load them. That is significant because we now catch any IO errors before spawning threads. Before we would leak threads if crashing in the middle.

Log example when trying to load such invalid segment:

2025-02-20T18:17:10.241651Z DEBUG collection::shards::local_shard: Segments path entry prefixed with a period, ignoring: ./storage/collections/embeddings_title_ada/0/segments/.DS_Store

All Submissions:

• Contributions should target the dev branch. Did you create your branch from dev?
• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

1. Does your submission pass tests?
2. Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
3. Have you checked your code using cargo clippy --all --all-features command?