Requirement

With the Jaeger storage extension for Jaeger-v2 is going to have full support of Jaeger-v1's storage backends, some unit tests on every storage backends are not enough. We need to conduct end-to-end tests of OpenTelemetry Collector pipeline to the targeted database.

Problem

There are still no integration tests to test the actual stored traces to the database from V2 Jaeger storage extension.

Proposal

Fortunately, OpenTelemetry Collector already has a testbed framework to help us conducting the end-to-end tests.

Testbed is a controlled environment and tools for conducting end-to-end tests for the OpenTelemetry Collector, including reproducible short-term benchmarks, correctness tests, long-running stability tests and maximum load stress tests. However, we will only utilize the correctness tests from testbed, it generates and sends every combinatorial trace attributes and matches every single of them with the received traces from another end.

Architecture of the integration test

Here's the architecture we will use to test the OpenTelemetry Collector pipeline from end-to-end with the designated storage backends.

Testbed components:

• LoadGenerator - encapsulates DataProvider and DataSender in order to generate and send data.
  • Golden DataProvider - generates traces from the "Golden" dataset generated using pairwise combinatorial testing techniques. Testbed example uses PICT to generate the test data, e.g. testdata.
  • OTLP Trace DataSender - with the generated traces from DataProvider, the DataSender sends traces to OTLP receiver in the collector instance.
• Mockbackend - encapsulates DataReceiver and provides consume functionality.
  • DataReceiver - we will create a custom DataReceiver that will host a Jaeger storage extension to retrieve traces from the database by pulling them using our artificial Jaeger storage receiver [jaeger-v2] Add support for artificial jaeger storage receiver #5242.
  • Consumer - consumer does not actually a thing in MockBackend but only to make the diagram intuitive, the traces received from our artificial receiver will be stored inside MockBackend.
• Correctness Test Validator - checks if the traces received from MockBackend are all matches with the generated traces from DataProvider.

Plan

The execution of integration tests will be done incrementally one by one on every supported storage backends:

• GRPC with memory
  • [jaeger-v2] add GRPC storage backend integration test #5259
  • [jaeger-v2] Build jaeger-remote-storage for gRPC integration test #5266
  • [jaeger-v2] add gRPC integration e2e test mode #5322
• Badger
  • [jaeger-v2] add Badger storage backend integration test #5281
  • [jaeger-v2] add badger e2e integration test #5355
• Cassandra [jaeger-v2] add cassandra e2e integration tests #5398
• Elasticsearch
  • Add Purge method for ES/OS #5407
  • [jaeger-v2] add elasticsearch & opensearch e2e integration test #5345
• Opensearch [jaeger-v2] add elasticsearch & opensearch e2e integration test #5345

Open questions

No response