fix: consider EC shard count in volume.balance capacity calculation #7034

FQHSLycopene · 2025-07-24T02:53:23Z

What problem are we solving?

Related Issues: Fixes #7033

When using volume.balance command in clusters with erasure coding (EC) volumes enabled, the operation fails with "no space left" error. This occurs because the balance strategy's capacity calculation doesn't account for EC shard usage, leading to incorrect load ratio calculations.

The capacityByMaxVolumeCount function only returns MaxVolumeCount without considering EC volume usage:

func capacityByMaxVolumeCount(diskType types.DiskType) CapacityFunc {
    return func(info *master_pb.DataNodeInfo) float64 {
        diskInfo, found := info.DiskInfos[string(diskType)]
        if !found {
            return 0
        }
        return float64(diskInfo.MaxVolumeCount)  // Only considers MaxVolumeCount
    }
}

This causes nodes with many EC volumes to be misjudged as "low load" and selected as targets for volume movement, even though they actually have no remaining capacity.

How are we solving the problem?

Added a new capacityByMaxVolumeCountWithEc function that properly accounts for EC shard usage:

func capacityByMaxVolumeCountWithEc(diskType types.DiskType) CapacityFunc {
    return func(info *master_pb.DataNodeInfo) float64 {
        diskInfo, found := info.DiskInfos[string(diskType)]
        if !found {
            return 0
        }
        var ecShardCount int
        for _, ecShardInfo := range diskInfo.EcShardInfos {
            ecShardCount += erasure_coding.ShardBits(ecShardInfo.EcIndexBits).ShardIdCount()
        }
        return float64(diskInfo.MaxVolumeCount) - float64(ecShardCount)/erasure_coding.DataShardsCount
    }
}

Changes Made:

Added new capacity function capacityByMaxVolumeCountWithEc that subtracts EC shard usage from available capacity
Updated balanceSelectedVolume to use the new capacity function
Maintained backward compatibility - the original capacityByMaxVolumeCount function remains unchanged

How is the PR tested?

Checks

I have added unit tests if possible.
I will add related wiki document changes and link to this PR after merging.

gemini-code-assist

Summary of Changes

Hello @FQHSLycopene, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request resolves a critical issue in the volume.balance command where its capacity calculations failed to properly account for erasure-coded (EC) volume shards, leading to erroneous 'no space left' errors and inefficient balancing. By introducing a new, more accurate capacity calculation function that considers EC shard usage, the system can now correctly assess node load and perform volume balancing effectively, ensuring better resource utilization and preventing unexpected failures.

Highlights

Problem Solved: Addresses the 'no space left' error encountered during volume.balance operations on clusters utilizing erasure-coded (EC) volumes. This error stemmed from an inaccurate capacity calculation that failed to account for EC shard usage, leading to nodes with many EC volumes being incorrectly identified as having low load.
New Capacity Calculation Function: Introduced a new function, capacityByMaxVolumeCountWithEc, which accurately calculates available capacity by subtracting the space consumed by EC shards from the maximum volume count. This provides a more realistic assessment of a data node's available capacity.
Integration and Backward Compatibility: The balanceSelectedVolume function has been updated to use the new capacityByMaxVolumeCountWithEc function for determining node capacity during balancing. The original capacityByMaxVolumeCount function remains unchanged, ensuring backward compatibility.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in issue comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments or fill out our survey to provide feedback.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request fixes a bug in the volume balancing logic for clusters with erasure coding by introducing a new capacity function that accounts for EC shards. The changes are logical and well-contained. A division-by-zero check has been added to improve the new function's robustness. It would also be beneficial to add unit tests for the new capacityByMaxVolumeCountWithEc function to cover various scenarios, including those with and without EC shards.

weed/shell/command_volume_balance.go

chrislusf · 2025-07-24T03:40:28Z

function capacityByMaxVolumeCount is used in 2 places. Instead of adding a new function capacityByMaxVolumeCountWithEc, how about changing the capacityByMaxVolumeCount implementation?

chrislusf · 2025-07-24T04:15:02Z

actually, there is already capacityByFreeVolumeCount function.

FQHSLycopene · 2025-07-25T01:10:11Z

Thank you for the suggestion!
The capacityByFreeVolumeCount function calculates the free volume count, but here we need the max volume count to calculate the ratio. I noticed that in other places where capacityByMaxVolumeCount is used, switching to the logic of capacityByMaxVolumeCountWithEc would also work as expected.
So I will update the implementation of capacityByMaxVolumeCount to include the EC shard usage, as you suggested.

…EC shard usage

fix: consider EC shard count in volume.balance capacity calculation

2a557f3

gemini-code-assist bot reviewed Jul 24, 2025

View reviewed changes

weed/shell/command_volume_balance.go Show resolved Hide resolved

update the implementation of capacityByMaxVolumeCount to include the …

e3a750a

…EC shard usage

chrislusf merged commit e1f8da0 into seaweedfs:master Jul 28, 2025
22 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix: consider EC shard count in volume.balance capacity calculation #7034

fix: consider EC shard count in volume.balance capacity calculation #7034

Uh oh!

FQHSLycopene commented Jul 24, 2025

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

gemini-code-assist bot left a comment

Uh oh!

Uh oh!

chrislusf commented Jul 24, 2025 •

edited

Loading

Uh oh!

chrislusf commented Jul 24, 2025

Uh oh!

FQHSLycopene commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fix: consider EC shard count in volume.balance capacity calculation #7034

fix: consider EC shard count in volume.balance capacity calculation #7034

Uh oh!

Conversation

FQHSLycopene commented Jul 24, 2025

What problem are we solving?

How are we solving the problem?

How is the PR tested?

Checks

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

chrislusf commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chrislusf commented Jul 24, 2025

Uh oh!

FQHSLycopene commented Jul 25, 2025

Uh oh!

Uh oh!

Uh oh!

chrislusf commented Jul 24, 2025 •

edited

Loading