[jaeger-v2] add gRPC integration e2e test mode (#5322)

## Which problem is this PR solving?
- Part of
#5254 (comment)
solves (1) and (4) for gRPC storage integration test in e2e mode.

## Description of the changes
- Utilizing existing `StorageIntegration` to test the jaeger-v2 OTel
Collector and gRPC storage backend with the provided config file at
`cmd/jaeger/grpc_config.yaml`.
- Creates an abstraction for e2e test mode that initializes the
collector, span writer to the receiver, and span reader from
jaeger_query.

## How was this change tested?
- Run `STORAGE=grpc SPAN_STORAGE_TYPE=memory make
jaeger-v2-storage-integration-test`

## Checklist
- [x] I have read
https://github.com/jaegertracing/jaeger/blob/master/CONTRIBUTING_GUIDELINES.md
- [x] I have signed all commits
- [x] I have added unit tests for the new functionality
- [x] I have run lint and test steps successfully
  - for `jaeger`: `make lint test`
  - for `jaeger-ui`: `yarn lint` and `yarn test`

---------

Signed-off-by: James Ryans <james.ryans2012@gmail.com>
Co-authored-by: Yuri Shkuro <yurishkuro@users.noreply.github.com>

.codecov.yml

@@ -11,7 +11,6 @@ ignore:
   - "thrift-gen/*/*"
   - "**/thrift-0.9.2/*"
   - "**/main.go"
-  - "cmd/jaeger/internal/integration/datareceivers"
   - "examples/hotrod"
 
 coverage:

.github/workflows/ci-grpc.yml

@@ -37,10 +37,13 @@ jobs:
       run: |
         case ${{ matrix.version }} in
           v1)
-            make grpc-storage-integration-test
+            SPAN_STORAGE_TYPE=memory \
+              make grpc-storage-integration-test
             ;;
           v2)
-            bash scripts/grpc-integration-test.sh latest
+            STORAGE=grpc \
+              SPAN_STORAGE_TYPE=memory \
+              make jaeger-v2-storage-integration-test
             ;;
         esac
 

Makefile

@@ -118,8 +118,9 @@ all-in-one-integration-test:
 # The integration tests are filtered by STORAGE env,
 # currently the available STORAGE variable is:
 #  - grpc
-.PHONY: jaeger-storage-integration-test
-jaeger-storage-integration-test:
+.PHONY: jaeger-v2-storage-integration-test
+jaeger-v2-storage-integration-test:
+	(cd cmd/jaeger/ && go build .)
 	# Expire tests results for jaeger storage integration tests since the environment might change
 	# even though the code remains the same.
 	go clean -testcache

cmd/jaeger/grpc_config.yaml

@@ -9,17 +9,13 @@ service:
 extensions:
   jaeger_query:
     trace_storage: external-storage
-    trace_storage_archive: external-storage-archive
     ui_config: ./cmd/jaeger/config-ui.json
 
   jaeger_storage:
     grpc:
       external-storage:
         server: localhost:17271
         connection-timeout: 5s
-      external-storage-archive:
-        server: localhost:17281
-        connection-timeout: 5s
 
 receivers:
   otlp:

cmd/jaeger/internal/integration/README.md

@@ -1,27 +1,42 @@
 # Integration
 
-Jaeger v2 integration tests are built on top of [OTEL Testbed module](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/testbed). OTEL Testbed provide comprehensive tools for conducting end-to-end tests for the OTEL Collector, such as reproducible short-term benchmarks, correctness tests, long-running stability tests and maximum load stress tests. However, we only utilize the correctness tests from testbed, it generates and sends every combinatorial trace attributes and matches every single of them with the received traces from another end. To learn more about OTEL Testbed, please refer to the their [README](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/testbed/README.md).
+The Jaeger v2 integration test is an extension of the existing `integration.StorageIntegration` designed to test the Jaeger-v2 OtelCol binary; currently, it only tests the span store. The existing tests at `plugin/storage/integration` (also called "unit mode") test by writing and reading span data directly to the storage API. In contrast, these tests (or "e2e mode") read and write span data through the RPC client to the Jaeger-v2 OtelCol binary. E2E mode tests read from the jaeger_query extension and write to the receiver in OTLP formats. For details, see the [Architecture](#architecture) section below.
 
 ## Architecture
 
-Here's the architecture to test the OpenTelemetry Collector pipeline from end-to-end with the designated storage backends.
-![integration diagram](integration-diagram.png)
+```mermaid
+flowchart LR
+    Test -->|writeSpan| SpanWriter
+    SpanWriter --> RPCW[RPC_client]
+    RPCW --> Receiver
+    Receiver --> Exporter
+    Exporter --> B(StorageBackend)
+    Test -->|readSpan| SpanReader
+    SpanReader --> RPCR[RPC_client]
+    RPCR --> jaeger_query
+    jaeger_query --> B
 
-Testbed components:
-| Component | Description |
-|-----------|-------------|
-| **LoadGenerator** | Encapsulates DataProvider and DataSender in order to generate and send data. |
-| Golden DataProvider | Generates traces from the "Golden" dataset generated using pairwise combinatorial testing techniques. Testbed example uses [PICT](https://github.com/microsoft/pict/) to generate the test data, e.g. [testdata](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/internal/coreinternal/goldendataset/testdata). |
-| OTLP Trace DataSender | With the generated traces from DataProvider, the DataSender sends traces to OTLP receiver in the collector instance. |
-| **Mockbackend** | Encapsulates DataReceiver and provides consume functionality. |
-| DataReceiver | A custom DataReceiver that will host a Jaeger storage extension to retrieve traces from the database by pulling them using our artificial Jaeger storage receiver. |
-| Consumer | Consumer does not actually a thing in MockBackend but only to make the diagram intuitive, the traces received from our artificial receiver will be stored inside MockBackend. |
-| **Correctness Test Validator** | Checks if the traces received from MockBackend are all matches with the generated traces from DataProvider. |
+    subgraph Integration Test Executable
+        Test
+        SpanWriter
+        SpanReader
+        RPCW
+        RPCR
+    end
+
+    subgraph jaeger-v2
+        Receiver
+        Exporter
+        jaeger_query
+    end
+```
 
 ## gRPC Integration Test
 
 To conduct the tests, run the following command:
 
 ```
-scripts/grpc-integration-test.sh <remote_storage_image_version>
+STORAGE=grpc \
+  SPAN_STORAGE_TYPE=memory \
+  make jaeger-v2-storage-integration-test
 ```