fix: Existing soft-deleted schema can be hard-deleted #26574

stuartcarnie · 2025-07-01T06:41:23Z

Helps

Allow users to update the hard-delete time of a soft-deleted schema object #26568

This PR addresses the following TODO items from #26568

Update DELETE API: Allow users to update the hard_delete_time of a database that has already been deleted
Improve Log Messages: Update log output to display correctly and include the hard_delete_time information, when specified

Further, it extends the databases and tables system tables for the _internal database to include a new column for the hard_deletion_date:

influxdb3 query --database _internal 'select * from system.tables'

+---------------+--------------------+--------------+--------------------+------------------+----------------------+---------+---------------------+
| database_name | table_name         | column_count | series_key_columns | last_cache_count | distinct_cache_count | deleted | hard_deletion_date  |
+---------------+--------------------+--------------+--------------------+------------------+----------------------+---------+---------------------+
| dev           | t1-20250701T063942 | 9            | tag0, tag1, tag2   | 0                | 0                    | true    | 2025-07-04T06:39:42 |
+---------------+--------------------+--------------+--------------------+------------------+----------------------+---------+---------------------+

If the schema is only soft-deleted, hard_deletion_date will be NULL.

Improved Log Messages

Shows explicitly the parameters used to delete the schema, such as in this case where --hard-delete now was specified:

2025-07-01T06:42:16.420899Z  INFO influxdb3_catalog::catalog::update: Delete database. db_name=dev-20250701T064143 db_id=1 hard_delete_time=now

and correctly shows Delete table, when deleting a table:

2025-07-01T06:39:42.680881Z  INFO influxdb3_catalog::catalog::update: Delete table. db_name=dev db_id=1 table_name=t1 table_id=2 hard_delete_time=default

…behavior - Renamed delete_table_defaults_to_hard_delete_never to delete_table_defaults_to_hard_delete_default - Renamed delete_database_defaults_to_hard_delete_never to delete_database_defaults_to_hard_delete_default - Updated assertions to expect default deletion duration instead of None/Never - Aligns with the change of HardDeletionTime default from Never to Default 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

praveen-influx

Looks good, extra comments in the tests are useful to follow the functionality. 👍

I left a minor query to understand the expected behavior.

praveen-influx · 2025-07-01T08:27:03Z

influxdb3_catalog/src/catalog/update.rs

+                hard_delete_time.as_time(&self.time_provider, self.default_hard_delete_duration());
+
+            let hard_delete_changed = db.hard_delete_time != hard_deletion_time;
+            if db.deleted && !hard_delete_changed {


Could it end up always missing this branch because we always set the hard delete time to default (HardDeletionTime::Default is new "default") when it's missing in request? That is maybe the expected behavior here, just wanted to clarify that.

I think a reasonable choice here is that if the value of the hard_delete_time parameter is Default and db.hard_delete_time is already set, it is also "not changed".

Latest commit addresses idempotence when a database or table is deleted using the default value for the hard deletion date. It will no longer keep updating the hard_deleted_date.

* Ensure system databases and tables schema specify a timezone for the `hard_deletion_time` Timestamp columns (otherwise they display without a timezone) * `DELETE` using `default` delay is idempotent, so multiple requests will not continue to update the `hard_deletion_time` * Improved test coverage for these behaviours

stuartcarnie · 2025-07-02T01:34:02Z

9a8ba51 improvements:

Ensure system databases and tables schema specify a timezone for the hard_deletion_time Timestamp columns (otherwise they display without a timezone)
DELETE using default delay is idempotent, so multiple requests will not continue to update the hard_deletion_time
- This addresses @praveen-influx's question
Improved test coverage for these behaviours

stuartcarnie · 2025-07-02T01:40:18Z

@praveen-influx note that amended the new hard_deletion_time column in the latest commit, so that it specifies the UTC timezone:

influxdb/influxdb3_server/src/system_tables/tables.rs

Lines 35 to 39 in 9a8ba51

    
           Field::new( 
        
               "hard_deletion_time", 
        
               DataType::Timestamp(TimeUnit::Second, Some(DEFAULT_TIMEZONE.into())), 
        
               true, 
        
           ),

This ensures that queries return a timezone specifier:

influxdb3 query --database _internal 'select database_name, table_name, hard_deletion_time from system.tables'

+---------------+--------------------+----------------------+
| database_name | table_name         | hard_deletion_time   |
+---------------+--------------------+----------------------+
| dev           | t1-20250702T011507 | 2025-07-05T01:15:01Z |
+---------------+--------------------+----------------------+

or as JSON:

[
  {
    "database_name": "dev",
    "table_name": "t1-20250702T011507",
    "hard_deletion_time": "2025-07-05T01:15:01Z"
  }
]

praveen-influx

LGTM

* chore: version bump in Cargo.toml to 3.3.0-nightly (#26566) * chore: version bump in Cargo.toml to 3.3.0-nightly * chore: update install script to point to 3.2.0 version * chore: Update to Rust 1.88 (#26567) * chore: Update to Rust 1.88 * chore: Fixes missed by Claude * fix: Add help text for the new update subcommand (#26569) This adds the update command to the help text output which was an oversight from my work in #26520. * fix: --object-store is explicitly marked required (#26575) Before this commit, although `--object-store` is mandatory it is not reflected in the error messages. The examples are listed in the issue 976. This commit makes `object_store` explicitly required which means error messages include `--object-store` listed as mandatory closes: influxdata/influxdb_pro#976 * chore: updates to Dockerfile (#26576) - update rust version to 1.88 - remove defaulting to `file` for object store helps: influxdata/influxdb_pro#976 * fix: v1 query API should default to ns for CSV output (#26577) * fix: Existing soft-deleted schema can be hard-deleted (#26574) * feat: Allow hard_deleted date of deleted schema to be updated * feat: Include hard_deletion_date in `_internal` `databases` and `tables` * feat: Unit tests for testing deleted databases * chore: Default is now to hard-delete with default duration * test: Update test names and assertions for new default hard deletion behavior - Renamed delete_table_defaults_to_hard_delete_never to delete_table_defaults_to_hard_delete_default - Renamed delete_database_defaults_to_hard_delete_never to delete_database_defaults_to_hard_delete_default - Updated assertions to expect default deletion duration instead of None/Never - Aligns with the change of HardDeletionTime default from Never to Default 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com> * chore: Remove TODO * chore: PR feedback and other improvements * Ensure system databases and tables schema specify a timezone for the `hard_deletion_time` Timestamp columns (otherwise they display without a timezone) * `DELETE` using `default` delay is idempotent, so multiple requests will not continue to update the `hard_deletion_time` * Improved test coverage for these behaviours --------- Co-authored-by: Claude <noreply@anthropic.com> --------- Co-authored-by: Stuart Carnie <stuart.carnie@gmail.com> Co-authored-by: Michael Gattozzi <mgattozzi@influxdata.com> Co-authored-by: wayne <wayne.warren.s@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>

stuartcarnie added 4 commits July 1, 2025 14:35

feat: Allow hard_deleted date of deleted schema to be updated

3cd7aaf

feat: Include hard_deletion_date in _internal databases and tables

628dccd

feat: Unit tests for testing deleted databases

9ee85d3

chore: Default is now to hard-delete with default duration

6f9e228

stuartcarnie requested a review from a team July 1, 2025 06:44

stuartcarnie self-assigned this Jul 1, 2025

stuartcarnie and others added 2 commits July 1, 2025 17:49

chore: Remove TODO

918528f

praveen-influx approved these changes Jul 1, 2025

View reviewed changes

praveen-influx approved these changes Jul 2, 2025

View reviewed changes

stuartcarnie merged commit 58b0725 into main Jul 2, 2025
12 checks passed

stuartcarnie deleted the sgc/26568/update_delete_api branch July 2, 2025 21:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix: Existing soft-deleted schema can be hard-deleted #26574

fix: Existing soft-deleted schema can be hard-deleted #26574

Uh oh!

stuartcarnie commented Jul 1, 2025 •

edited

Loading

Uh oh!

praveen-influx left a comment

Uh oh!

praveen-influx Jul 1, 2025

Uh oh!

stuartcarnie Jul 1, 2025

Uh oh!

stuartcarnie Jul 2, 2025

Uh oh!

stuartcarnie commented Jul 2, 2025

Uh oh!

stuartcarnie commented Jul 2, 2025

Uh oh!

praveen-influx left a comment

Uh oh!

Uh oh!

Uh oh!

fix: Existing soft-deleted schema can be hard-deleted #26574

fix: Existing soft-deleted schema can be hard-deleted #26574

Uh oh!

Conversation

stuartcarnie commented Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Improved Log Messages

Uh oh!

praveen-influx left a comment

Choose a reason for hiding this comment

Uh oh!

praveen-influx Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

stuartcarnie Jul 1, 2025

Choose a reason for hiding this comment

Uh oh!

stuartcarnie Jul 2, 2025

Choose a reason for hiding this comment

Uh oh!

stuartcarnie commented Jul 2, 2025

Uh oh!

stuartcarnie commented Jul 2, 2025

Uh oh!

praveen-influx left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

stuartcarnie commented Jul 1, 2025 •

edited

Loading