Exclude 8.8 fields from being serialized for 8.7 records #37346

panagiotisgts · 2025-08-28T14:36:10Z

Description

Conditionally exclude fields introduced in 8.8 for records with version 8.7 from being serialized in ES/OS Exporter exported records as it breaks schema mappings.

Tests had to be adjusted to have a specified Broker Version in the records generated so as to have a parsable semver value

Checklist

Enable backports when necessary (fex. for bug fixes or for CI changes).

Related issues

closes #37343

Exclude fields introduced in 8.8 from being part of the serialized ES/OS exported document for previous versions

houssain-barouni

Thank you for handling this 💪
I have added some comments and suggestions.

Let's consider creating a follow-up task to add a unit (revapi?) test ensuring all new fields are excluded when serializing records from previous versions.

I have not exhaustively compared all the 8.7 and 8.8 differences. Let me know if you would like me to take a closer look.

...porters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/BulkIndexRequest.java

houssain-barouni · 2025-08-29T13:35:46Z

...porters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/BulkIndexRequest.java

+    PROCESS_DEFINITION_PATH_PROPERTY,
+    CALLING_ELEMENT_PATH_PROPERTY,
+  })
+  private static final class ProcessInstanceMixin {}


same here ProcessInstance87Mixin ?

Mixins would most likely be used for the previous versions, not sure whether we'd to make this explicit

EvaluatedDecisionMixin is used for the current version, so it can be confusing

...porters/elasticsearch-exporter/src/main/java/io/camunda/zeebe/exporter/BulkIndexRequest.java

houssain-barouni

LGTM! 🚀
To avoid confusion, I still recommend including the version in the names of the Mixin classes.

I will add backport labels to backport this to stable/8.7 and 8.6 to fix #37283.
Some adaptation will be required for these branches. If you are busy with other tasks, I can handle this. This issue is urgent, so it would be beneficial to have the fix included in the upcoming patch releases.

For the backport to release-8.8.0-alpha8, you need to check with @cmur2 , if we can still have this in the alpha8 release.

github-actions · 2025-09-01T12:56:09Z

🧪 Flaky Tests Summary

👻 Haunted Tests — They Fail When No One's Watching

shouldAssign87ZeebeTaskV1 – 💀 100% flakiness
- Jobs: opensearch-integration-tests
- Package: io.camunda.it.migration.usertask
- Class: AssignUserTaskMigrationIT
- Occurrences: 1 / 1
shouldMigrateProcess – 💀 100% flakiness
- Jobs: opensearch-integration-tests
- Package: io.camunda.it.orchestration
- Class: ProcessMigrationIT
- Occurrences: 1 / 1

If the changes affect this area, please check and fix before merging.
If not, leave a quick comment explaining why it's likely unrelated.

backport-action · 2025-09-01T14:17:22Z

Created backport PR for stable/8.6:

Exclude 8.6 fields from being serialized for 8.5 records #37462 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-37346-to-stable/8.6
git worktree add --checkout .worktree/backport-37346-to-stable/8.6 backport-37346-to-stable/8.6
cd .worktree/backport-37346-to-stable/8.6
git reset --hard HEAD^
git cherry-pick -x f1b7a665a94d15300d9cc07b6a7a9487e564a9b3 cc4f633fd2c43a82397824340640c99fcd943d44 2d4e2fc021a5ac3911ff7c3d8582faf701161b03 bfa38c811a854ac1b13eef6b8c99f13319b3bc7e
git push --force-with-lease

backport-action · 2025-09-01T14:17:27Z

Created backport PR for stable/8.7:

Exclude 8.7 fields from being serialized for 8.6 records #37463 with remaining conflicts!

Please cherry-pick the changes locally and resolve any conflicts.

git fetch origin backport-37346-to-stable/8.7
git worktree add --checkout .worktree/backport-37346-to-stable/8.7 backport-37346-to-stable/8.7
cd .worktree/backport-37346-to-stable/8.7
git reset --hard HEAD^
git cherry-pick -x f1b7a665a94d15300d9cc07b6a7a9487e564a9b3 cc4f633fd2c43a82397824340640c99fcd943d44 2d4e2fc021a5ac3911ff7c3d8582faf701161b03 bfa38c811a854ac1b13eef6b8c99f13319b3bc7e
git push --force-with-lease

backport-action · 2025-09-01T14:17:28Z

other side closed

backport-action · 2025-09-01T14:17:34Z

Successfully created backport PR for release-8.8.0-alpha8:

[Backport release-8.8.0-alpha8] Exclude 8.8 fields from being serialized for 8.7 records #37464

…zed for 8.7 records (#37464) # Description Backport of #37346 to `release-8.8.0-alpha8`. relates to #37343

# Description Backport of #37346 to `stable/8.7`. closes #37283 relates to #37343 8.6 vs 8.7 diffs show ``` > diff -r 8.7/zeebe/exporters/elasticsearch-exporter/src/main/resources 8.6/zeebe/exporters/elasticsearch-exporter/src/main/resources diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-deployment-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-deployment-template.json 161,191d160 < "resourceMetadata": { < "properties": { < "resourceId": { < "type": "keyword" < }, < "version": { < "type": "long" < }, < "resourceKey": { < "type": "long" < }, < "resourceName": { < "type": "text" < }, < "checksum": { < "enabled": false < }, < "duplicate": { < "type": "boolean" < }, < "tenantId": { < "type": "keyword" < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" < } < } < }, ```

# Description Backport of #37346 to `stable/8.6`. closes #37283 relates to #37343 8.5 vs 8.6 diffs: ``` diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-command-distribution-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-command-distribution-template.json 25,27d24 < "queueId": { < "type": "keyword" < }, diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-decision-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-decision-template.json 45,50d44 < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-deployment-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-deployment-template.json 46,51d45 < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" 121,126d114 < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" 152,157d139 < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" 163,165d144 < }, < "deploymentKey": { < "type": "long" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-form-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-form-template.json 47,52d46 < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-incident-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-incident-template.json 51,59d50 < }, < "elementInstancePath": { < "enabled": false < }, < "processDefinitionPath": { < "enabled": false < }, < "callingElementPath": { < "enabled": false diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-job-batch-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-job-batch-template.json 97,102d96 < }, < "jobListenerEventType": { < "type": "keyword" < }, < "changedAttributes": { < "type": "text" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-job-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-job-template.json 78,83d77 < }, < "jobListenerEventType": { < "type": "keyword" < }, < "changedAttributes": { < "type": "text" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-process-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-process-template.json 45,50d44 < }, < "deploymentKey": { < "type": "long" < }, < "versionTag": { < "type": "keyword" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-template.json 53,55d52 < }, < "operationReference": { < "type": "long" diff -r zeebe/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-user-task-template.json zeebe3/zeebe/exporters/elasticsearch-exporter/src/main/resources/zeebe-record-user-task-template.json 81,83d80 < }, < "priority": { < "type": "integer" ```

fix: exclude batchOperationReference for 8.7 records from being exported

f1b7a66

github-actions bot added the component/zeebe Related to the Zeebe component/team label Aug 28, 2025

fix: exclude rest of the 8.8 fields

cc4f633

Exclude fields introduced in 8.8 from being part of the serialized ES/OS exported document for previous versions

panagiotisgts requested review from ChrisKujawa, a team and houssain-barouni and removed request for a team August 29, 2025 13:16

panagiotisgts changed the title ~~fix: exclude batchOperationReference for 8.7 records from being exported~~ Exclude 8.8 fields from being serialized for 8.7 records Aug 29, 2025

panagiotisgts added the backport release-8.8.0-alpha8 label Aug 29, 2025

houssain-barouni requested a review from a team August 29, 2025 13:32

houssain-barouni reviewed Aug 29, 2025

View reviewed changes

panagiotisgts requested a review from houssain-barouni September 1, 2025 11:56

fix: address review comments

2d4e2fc

panagiotisgts force-pushed the pg/37343-exclude-batchOperationReference branch from 5d22749 to 2d4e2fc Compare September 1, 2025 12:05

houssain-barouni approved these changes Sep 1, 2025

View reviewed changes

houssain-barouni added backport stable/8.6 Backport a pull request to stable/8.6 backport stable/8.7 Backport a pull request to stable/8.7 labels Sep 1, 2025

refactor: version tag mixins

bfa38c8

panagiotisgts force-pushed the pg/37343-exclude-batchOperationReference branch from 5c23ab6 to bfa38c8 Compare September 1, 2025 12:33

panagiotisgts added this pull request to the merge queue Sep 1, 2025

github-merge-queue bot removed this pull request from the merge queue due to failed status checks Sep 1, 2025

houssain-barouni added this pull request to the merge queue Sep 1, 2025

Merged via the queue into main with commit b62c0ae Sep 1, 2025
105 checks passed

houssain-barouni deleted the pg/37343-exclude-batchOperationReference branch September 1, 2025 14:17

backport-action mentioned this pull request Sep 1, 2025

Exclude 8.6 fields from being serialized for 8.5 records #37462

Merged

backport-action mentioned this pull request Sep 1, 2025

Exclude 8.7 fields from being serialized for 8.6 records #37463

Merged

backport-action mentioned this pull request Sep 1, 2025

[Backport release-8.8.0-alpha8] Exclude 8.8 fields from being serialized for 8.7 records #37464

Merged

panagiotisgts added a commit that referenced this pull request Sep 2, 2025

[Backport release-8.8.0-alpha8] Exclude 8.8 fields from being seriali…

821f258

…zed for 8.7 records (#37464) # Description Backport of #37346 to `release-8.8.0-alpha8`. relates to #37343

github-actions bot added version:8.8.0-alpha8-rc3 version:8.7.12 labels Sep 2, 2025

github-actions bot added version:8.6.26 version:8.8.0-alpha8 labels Sep 2, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Exclude 8.8 fields from being serialized for 8.7 records #37346

Exclude 8.8 fields from being serialized for 8.7 records #37346

Uh oh!

panagiotisgts commented Aug 28, 2025 •

edited

Loading

Uh oh!

houssain-barouni left a comment

Uh oh!

Uh oh!

Uh oh!

houssain-barouni Aug 29, 2025

Uh oh!

panagiotisgts Sep 1, 2025

Uh oh!

houssain-barouni Sep 1, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

houssain-barouni left a comment •

edited

Loading

Uh oh!

github-actions bot commented Sep 1, 2025

Uh oh!

Uh oh!

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

Uh oh!

Exclude 8.8 fields from being serialized for 8.7 records #37346

Exclude 8.8 fields from being serialized for 8.7 records #37346

Uh oh!

Conversation

panagiotisgts commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Related issues

Uh oh!

houssain-barouni left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

houssain-barouni Aug 29, 2025

Choose a reason for hiding this comment

Uh oh!

panagiotisgts Sep 1, 2025

Choose a reason for hiding this comment

Uh oh!

houssain-barouni Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

houssain-barouni left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 1, 2025

🧪 Flaky Tests Summary

Uh oh!

Uh oh!

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

backport-action commented Sep 1, 2025

Uh oh!

Uh oh!

panagiotisgts commented Aug 28, 2025 •

edited

Loading

houssain-barouni Sep 1, 2025 •

edited

Loading

houssain-barouni left a comment •

edited

Loading