This pull request introduces significant changes to the seed peer management and preheating logic in the scheduler, primarily aimed at improving flexibility, simplifying the codebase, and enhancing the selection of seed peers. The most notable changes include replacing the centralized SeedPeerClient with a distributed client selection approach, introducing a hash ring for seed peer selection, and updating the HostManager to distinguish between normal and non-normal hosts.

Seed Peer Management Enhancements:

• Replaced the SeedPeerClient with a distributed client selection mechanism, removing the centralized client implementation (scheduler/resource/standard/seed_peer_client.go removed entirely).
• Introduced a hash ring for seed peer selection based on task IDs, enabling consistent and balanced seed peer assignment (scheduler/resource/standard/seed_peer.go).
• Updated the SeedPeer interface to include a SelectSeedPeer method, replacing the old Client method for selecting seed peers dynamically (scheduler/resource/standard/seed_peer.go).

Host Management Updates:

• Added a new method LoadAllNonNormals to HostManager to retrieve non-normal hosts (used to identify seed peers), and implemented this method in scheduler/resource/standard/host_manager.go. [1] [2]
• Updated the mock implementation of HostManager to support the new LoadAllNonNormals method (scheduler/resource/standard/host_manager_mock.go).

Preheating Logic Improvements:

• Updated the preheating logic to use the newly introduced SelectSeedPeer method for selecting seed peers dynamically, replacing the previous approach of relying on a static client (scheduler/job/job.go). [1] [2] [3]
• Modified the preheatV1SingleSeedPeer and preheatV2SingleSeedPeerByURL methods to create gRPC clients dynamically for the selected seed peer, with a note to reuse clients in the future if performance issues arise (scheduler/job/job.go). [1] [2]

Codebase Simplification:

• Removed the SeedPeerClient implementation and its associated methods, significantly simplifying the codebase (scheduler/resource/standard/seed_peer_client.go removed entirely).
• Updated the Resource initialization to directly create the SeedPeer without the intermediate SeedPeerClient (scheduler/resource/standard/resource.go).

These changes collectively improve the flexibility and maintainability of the seed peer management system, while also laying the groundwork for better scalability and performance in future iterations.… self pick

Description

Related Issue

#4217

Motivation and Context

Screenshots (if appropriate)

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Documentation Update (if none of the other choices apply)

Checklist

• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have added tests to cover my changes.