Skip to content

[Feature Request]: Segment compaction #466

@small-turtle-1

Description

@small-turtle-1

Is there an existing issue for the same feature request?

  • I have checked the existing issues.

Is your feature request related to a problem?

1. Each import create a new segment and import data in new block in the new segment. The not filled block may waste disk space.
2. The compaction also remove delete row to save disk space.
3. The index is created in segment granulrity, small segment will degrade the performance of index.
4. The index rebuild is not solved in this issue

Describe the feature you'd like

A backend task scan the table in period, if segment can be merged then merge it.

  1. The merge create new segment and datablock. Apply greedy algorithm to choose to-merge segments. (This is np problem)
  2. If compacted segment is altered(only deleted here, because compacting segment is closed) in compact process, then do the alter log in new segment until no alter is made.
  3. Mark old segment as deprecated and commit the new segment to replace old.
  4. For frontend delete operation, when commit, check if the segment is deprecated. If so, abort.

Describe implementation you've considered

No response

Documentation, adoption, use case

No response

Additional information

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions