Features

• Schema Registry: Support granular authorization of endpoints via ACLs by @BenPope in #26758
• Support for JSON (via JSON Schema) translation to Iceberg. by @nvartolomei in #26271
• This adds support for using the AWS Glue Data Catalog as an Iceberg REST catalog. To use Glue as an Iceberg REST catalog, configure the standard Iceberg REST catalog and AWS cloud storage configuration and set iceberg_rest_catalog_authentication_mode to aws_sigv4. Additionally, the Glue Data Catalog requires a base location for table storage, configured by iceberg_rest_catalog_base_location. by @wdberkeley in #26363
• Add support for `max.compaction.lag.ms which sets the maximum time before a message becomes eligible for compaction. This can be used to induce periodic compaction of a topic in some cases. by @wdberkeley in #26172
• Add support for min.compaction.lag.ms. In a compacted topic, this property sets the minimum time for which a message will remain ineligible for compaction. This can be used to reduce the frequency of compactions in some cases. by @wdberkeley in #26172
• Added support for the format=serialized query parameter for protobuf schemas for the following schema registry endpoints: by @IoannisRP in #26124
  • get schema/ids/{id}
  • post subject/{sub}
  • get subject/{subject}/version/{version}
  • get subject/{subject}/version/{version}/schema
• Kafka: Implement KIP-700 DescribeCluster (API 60) v0 by @BenPope in #25919
• Redpanda now supports KIP-554 by @michael-redpanda in
  #24935
• adds the kafka_topics_max cluster configuration that can be used to enforce a maximum user topic count cluster-wide. by @ballard26 in #26302
• Allows direct uploading of debug bundles collected with rpk debug remote-bundle by @JFlath in #26399
• rpk: Get df output in debug bundle by @JFlath in
  #23753
• Adds the topic_label_aggregation_limit cluster config that when set will dynamically aggregate the topic label in metrics when topic count exceeds the set value. by @ballard26 in #26385
• Force reconfiguration/recovery of partitions now supports updating any in-progress move, regardless of its state. This makes it much easier to use as an escape hatch to recover partitions in various stuck scenarios that would otherwise be difficult to recover. by @bharathv in #26549
• Add the ability to "force" a node ID override in situations where the node already has an ID assigned [INC-774]. by @oleiman in
  #25963
• pandaproxy/sr: [Core-9938] Add support for format query parameter by @IoannisRP in
  #25971
• Introduces two new loggers, pandaproxy/requests and schemaregistry/requests for the printing of full request and response bodies of Rest Proxy (Pandaproxy) and Schema Registry requests when enabled at the trace level. by @pgellert in #25824
• allow use rpk cluster config get in Redpanda cloud clusters. by @andresaristizabal in #26133
• rpk security secret: Now you can modify the scope for which a Redpanda cloud secret is created. by @andresaristizabal in #25747

In addition, there are many improvements and bug fixes in this release.

Full Changelog: v25.1.1...v25.2.1

Features

• Force reconfiguration now supports updating any in-progress move, regardless of its state. This makes it much easier to use as an escape hatch to recover partitions in various stuck scenarios that would otherwise be difficult to recover. by @bharathv in #26669
• rpk cluster config list: new command to list all cluster configuration properties in the cluster by @andresaristizabal in #26825

Bug Fixes

• Fixed race condition in metadata auto create topics by @michael-redpanda in #26944
• Fixes a bug in which a partition for a tiered storage enabled topic could provide an incorrect response to an OffsetForLeaderEpoch request by @WillemKauf in #26880
• #26838 Fixes a bug in timequeries performed over local storage which could lead to inconsistent or undefined results. by @WillemKauf in #26843
• repair initial leadership appointment machinery by @joe-redpanda in #26852
• rpk: fixes a bug where issuing batched FindCoordinator requests consistently failed (used in rpk topic describe when describing multiple topics). by @r-vasquez in #26850

Improvements

• Per-partition cloud storage scrubber metrics will now appear only when the feature is turned on at cluster level (cloud_storage_enable_scrubbing == true) by @oleiman in #26846
• Redpanda will now validate that an Iceberg REST endpoint is set when cluster configuration is altered to enable the Iceberg REST catalog. by @wdberkeley in #26856
• rpk container now uses Redpanda Console version 3.1.2 by @r-vasquez in #26885
• PR #26692 [v25.1.x] storage: quiet logging when no tiered storage topics by @dotnwat
• PR #26848 [v25.1.x] iceberg/config: rename iceberg_rest_catalog_aws_credentials_source by @wdberkeley
• PR #26923 [v25.1.x] Fixes RandomNodeOpeartions decommission stopped making progress test by @mmaslankaprv
• PR #26935 [v25.1.x] rptest: remove test_create_or_delete_topics_while_busy by @nvartolomei

Full Changelog: v25.1.8...v25.1.9

Bug Fixes

• Fixed an issue with consumer groups with manually assigned consumers. When an OffsetDeleteRequest was sent on such a group, a GROUP_SUBSCRIBED_TO_TOPIC error was returned. by @IoannisRP in #26705
• Increase the default self check timeout from 5s to 10s to leave time to retry DNS lookups if they time out during a self check operation. by @pgellert in #26775
• Resolves a memory leak scenario in node_status_backend by resetting connections which make no progress by @joe-redpanda in #26802
• Return the correct error response if the RPC to the leader for deleting ACLs fails. by @BenPope in #26785
• This fixes a bug in Redpanda's self-check functionality, where the self-check would occasionally fail with 'Uploaded key/payload could not be found in cloud storage item list.' despite the object being successfully uploaded. This issue occurred when testing against an Azure ABS tiered storage endpoint. by @pgellert in #26728
• #26739 Fixes a bug in which a segment produced by adjacent merge compaction did not have its batch cache reset, leading to potentially stale reads in the storage layer. by @WillemKauf in #26741
• #26820 Fixes a bug where data loss could occur during FPM w/ tiered storage disabled by @oleiman in #26821

Improvements

• Make segment download timeouts configurable in cloud cache hydration by @oleiman in #26779
• PR #26744 [v24.3.x] [CORE-12729] debug/bundle: forward kubernetes env vars to rpk by @IoannisRP
• PR #26815 [v24.3.x] [CORE-8805] dt/archival: Decrease manifest upload interval to avoid race by @oleiman
• PR #26769 [v24.3.x] raft/heartbeat_manager: timeout memory leak fix by @joe-redpanda

Full Changelog: v24.3.17...v24.3.18

Bug Fixes

• Resolves a memory leak scenario in node_status_backend by resetting connections which make no progress by @joe-redpanda in #26801
• Return the correct error response if the RPC to the leader for deleting ACLs fails. by @BenPope in #26784
• #26819 Fixes a bug where data loss could occur during FPM w/ tiered storage disabled by @oleiman in #26822
• PR #26817 [v24.2.x] [CORE-8805] dt/archival: Decrease manifest upload interval to avoid race by @oleiman
• PR #26770 [v24.2.x] raft/heartbeat_manager: timeout memory leak fix by @joe-redpanda

Full Changelog: v24.2.26...v24.2.27

Bug Fixes

• Fixed an issue with consumer groups with manually assigned consumers. When an OffsetDeleteRequest was sent on such a group, a GROUP_SUBSCRIBED_TO_TOPIC error was returned. by @IoannisRP in #26704
• Fixes a bug where data loss could occur during FPM w/ tiered storage disabled by @oleiman in #26818
• Fixes a hang in RPC dispatch that may result in failed replication and leadership transfers. by @bharathv in #26805
• Increase the default self check timeout from 5s to 10s to leave time to retry DNS lookups if they time out during a self check operation. by @pgellert in #26776
• Resolves a memory leak scenario in node_status_backend by resetting connections which make no progress by @joe-redpanda in #26800
• Return the correct error response if the RPC to the leader for deleting ACLs fails. by @BenPope in #26783
• This fixes a bug in Redpanda's self-check functionality, where the self-check would occasionally fail with 'Uploaded key/payload could not be found in cloud storage item list.' despite the object being successfully uploaded. This issue occurred when testing against an Azure ABS tiered storage endpoint. by @pgellert in #26713
• #26738 Fixes a bug in which a segment produced by adjacent merge compaction did not have its batch cache reset, leading to potentially stale reads in the storage layer. by @WillemKauf in #26740

Improvements

• Fall back to the previously uploaded cluster manifest's group offset snapshot if uploading the group offsets fails for a consumer offsets topic partition. by @pgellert in #26793
• Make segment download timeouts configurable in cloud cache hydration by @oleiman in #26780
• PR #26641 [v25.1.x] [CORE-8392] http: Add shutdown connection error code by @Lazin
• PR #26699 [backport v25.1.x] iceberg/config: mark iceberg auth options as restored (default) by @wdberkeley
• PR #26734 [v25.1.x] [CORE-12729] debug/bundle: forward kubernetes env vars to rpk by @IoannisRP
• PR #26771 [v25.1.x] raft/heartbeat_manager: timeout memory leak fix by @joe-redpanda

Full Changelog: v25.1.7...v25.1.8

Bug Fixes

• Fixes unbounded memory usage in some transaction use caes by @bharathv in #26681

Improvements

• PR #26649 [v25.1.x] Backport AWS Glue REST catalog support by @wdberkeley
• PR #26666 [v25.1.x] Added defensive checks when materializing batch records by @mmaslankaprv
• PR #26674 [v25.1.x] c/rm_frontend: more nuanced mapping of error when locking writes by @mmaslankaprv
• PR #26701 [backport v25.1.x] datalake: add default partition spec kludge for AWS Glue by @wdberkeley

Full Changelog: v25.1.6...v25.1.7

Bug Fixes

• Fix Avro translation to Iceberg when root Avro schema is a primitive type. by @nvartolomei in #26461
• Fix an issue where audit log could lock down a cluster, if miss-configured. Now it is always possible to disable it. by @IoannisRP in #26652
• Iceberg integration: Encode avro record field names to avoid using disallowed characters (like dots with default partitioning hour(redpanda.timestamp). This makes it possible to read Iceberg data with latest version of DuckDB. by @nvartolomei in #26535
• prevents Redpanda from crashing when reading invalid record data by @mmaslankaprv in #26492

Improvements

• Adds support for the Iceberg table properties write.metadata.path and write.data.path. When an Iceberg catalog defines these properties, Redpanda will use them to determine where to write Iceberg table metadata and data, respectively, instead of using default locations based on the table location. by @wdberkeley in #26440
• Allows direct uploading of debug bundles collected with rpk debug remote-bundle by @JFlath in #26515
• Cut down the amount of time spent in fstat() syscalls during storage layer housekeeping & cut down the amount of time spent in fstat() syscalls in the storage layer EVEN MORE IN GENERAL! by @WillemKauf in #26656
• Fixed large allocation issues when handling OffsetCommits by @mmaslankaprv in #26414
• Fixes an issue in which users could experience oversized allocations during a DescribeGroup request. by @WillemKauf in #26532
• ability to control batch cache settings for __consumer_offsets topic by @mmaslankaprv in #26558

Full Changelog: v25.1.5...v25.1.6

Bug Fixes

• Fix an issue where audit log could lock down a cluster, if miss-configured. Now it is always possible to disable it. by @IoannisRP in #26651
• Fixes unbounded memory usage in some transaction use caes by @bharathv in #26682

Improvements

• ability to control batch cache settings for __consumer_offsets topic by @mmaslankaprv in #26658
• PR #25397 [v24.3.x] [CORE-8946] cloud_storage: Update process_anomalies method by @Lazin
• PR #26177 [v24.3.x] r/consensus: do not block leadership completely in maintenance mode by @mmaslankaprv
• PR #26526 [v24.3.x] ducktape: Respect rpk timeout in rpk by @StephanDollberg
• PR #26548 [v24.3.x] [CORE-12155] Introduce external timeout for cloud_storage client leases by @oleiman
• PR #26554 [v24.3.x] Fix race between bootstrap and shutdown by @bashtanov
• PR #26578 [v24.3.x] csc/client_pool: Add null checks in lease watchdog handler by @oleiman
• PR #26599 [v24.3.x] kc/consumer: fixed resource leak when coordinator changes by @mmaslankaprv
• PR #26638 [v24.3.x] raft: Handle exceptions in backgroun_apply_fiber by @Lazin
• PR #26640 [v24.3.x] [CORE-8392] http: Add shutdown connection error code by @Lazin
• PR #26665 [v24.3.x] Added defensive checks when materializing batch records by @mmaslankaprv

Full Changelog: v24.3.16...v24.3.17

Bug Fixes

• Fixes unbounded memory usage in some transaction use caes by @bharathv in #26683
• prevents Redpanda from crashing when reading invalid record data by @mmaslankaprv in #26493
• PR #26433 [v24.2.x] storage: fix index state truncate overflow by @andrwng

Full Changelog: v24.2.25...v24.2.26

Bug Fixes

• prevents Redpanda from crashing when reading invalid record data by @mmaslankaprv in #26494

Improvements

• Allows direct uploading of debug bundles collected with rpk debug remote-bundle by @JFlath in #26514
• Fixes an issue in which users could experience oversized allocations during a DescribeGroup request. by @WillemKauf in #26531
• PR #26444 [v24.3.x] storage: call reserve() in storage::range() by @WillemKauf

Full Changelog: v24.3.15...v24.3.16