1
0
mirror of https://github.com/open-telemetry/opentelemetry-go.git synced 2024-12-14 10:13:10 +02:00
opentelemetry-go/exporters/otlp/otlptrace/otlptracegrpc/mock_collector_test.go

180 lines
4.3 KiB
Go
Raw Normal View History

// Copyright The OpenTelemetry Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
package otlptracegrpc_test
import (
"context"
"fmt"
"net"
"sync"
"testing"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/internal/otlptracetest"
collectortracepb "go.opentelemetry.io/proto/otlp/collector/trace/v1"
tracepb "go.opentelemetry.io/proto/otlp/trace/v1"
)
func makeMockCollector(t *testing.T, mockConfig *mockConfig) *mockCollector {
return &mockCollector{
t: t,
traceSvc: &mockTraceService{
storage: otlptracetest.NewSpansStorage(),
errors: mockConfig.errors,
},
}
}
type mockTraceService struct {
collectortracepb.UnimplementedTraceServiceServer
Fix goroutine leaks in otlptracegrpc testing (#2409) * Fail in RunEndToEndTest if collector stop fails * Use testing T.Cleanup to check shut downs * Add goroutine leak detection * Fix TestExporterShutdown go leak The shutdown tests checking if a context error is honored did not completely clean up the resources used by the client after the error was evaluated. Update the connection client to handle multiple calls to shutdown and make a second call to these clients that must succeed so the test does not have abandoned goroutines. * Fix leak in TestNew_WithTimeout The mockTraceService did not delay with its lock being held. This resulted in the mockCollector stopping and being able to acquire the lock. It was assumed that no export was taking place because of this and the mockTraceService was abandoned without cleaning up resources it held and goroutines it had spawned. This reworks the export blocking logic to block on a channel read. This will make the block more deterministic and not depend on the scheduler timing. Additionally, this blocking is moved inside the lock acquire. Meaning code will deadlock if the block is not released before a shutdown (something the developer will immediately be aware of when they submit a bad patch), and will ensure all resources are released before shutdown. Replace TestNew_WithTimeout with TestExportSpansTimeoutHonored which directly tests if a span export errors when the timeout is reached. This is the only unique thing that TestNew_WithTimeout, but it also tests the non-error path. That non-error path is tested in many other tests. * Guard otlptracehttp client stopCh when stopping In normal operations the exporter is guaranteed to only ever call the client Stop method once. However in testing we need to call this multiple times when checking it returns an error in particular context. Add a lightweight sync.Once to the closing of the stopCh to ensure tests do not panic when cleaning up. * Release export block after export Prevent deadlock in TestExportSpansTimeoutHonored.
2021-11-22 17:54:32 +02:00
errors []error
requests int
mu sync.RWMutex
storage otlptracetest.SpansStorage
headers metadata.MD
exportBlock chan struct{}
}
func (mts *mockTraceService) getHeaders() metadata.MD {
mts.mu.RLock()
defer mts.mu.RUnlock()
return mts.headers
}
func (mts *mockTraceService) getSpans() []*tracepb.Span {
mts.mu.RLock()
defer mts.mu.RUnlock()
return mts.storage.GetSpans()
}
func (mts *mockTraceService) getResourceSpans() []*tracepb.ResourceSpans {
mts.mu.RLock()
defer mts.mu.RUnlock()
return mts.storage.GetResourceSpans()
}
func (mts *mockTraceService) Export(ctx context.Context, exp *collectortracepb.ExportTraceServiceRequest) (*collectortracepb.ExportTraceServiceResponse, error) {
mts.mu.Lock()
defer func() {
mts.requests++
mts.mu.Unlock()
}()
Fix goroutine leaks in otlptracegrpc testing (#2409) * Fail in RunEndToEndTest if collector stop fails * Use testing T.Cleanup to check shut downs * Add goroutine leak detection * Fix TestExporterShutdown go leak The shutdown tests checking if a context error is honored did not completely clean up the resources used by the client after the error was evaluated. Update the connection client to handle multiple calls to shutdown and make a second call to these clients that must succeed so the test does not have abandoned goroutines. * Fix leak in TestNew_WithTimeout The mockTraceService did not delay with its lock being held. This resulted in the mockCollector stopping and being able to acquire the lock. It was assumed that no export was taking place because of this and the mockTraceService was abandoned without cleaning up resources it held and goroutines it had spawned. This reworks the export blocking logic to block on a channel read. This will make the block more deterministic and not depend on the scheduler timing. Additionally, this blocking is moved inside the lock acquire. Meaning code will deadlock if the block is not released before a shutdown (something the developer will immediately be aware of when they submit a bad patch), and will ensure all resources are released before shutdown. Replace TestNew_WithTimeout with TestExportSpansTimeoutHonored which directly tests if a span export errors when the timeout is reached. This is the only unique thing that TestNew_WithTimeout, but it also tests the non-error path. That non-error path is tested in many other tests. * Guard otlptracehttp client stopCh when stopping In normal operations the exporter is guaranteed to only ever call the client Stop method once. However in testing we need to call this multiple times when checking it returns an error in particular context. Add a lightweight sync.Once to the closing of the stopCh to ensure tests do not panic when cleaning up. * Release export block after export Prevent deadlock in TestExportSpansTimeoutHonored.
2021-11-22 17:54:32 +02:00
if mts.exportBlock != nil {
// Do this with the lock held so the mockCollector.Stop does not
// abandon cleaning up resources.
<-mts.exportBlock
}
reply := &collectortracepb.ExportTraceServiceResponse{}
if mts.requests < len(mts.errors) {
idx := mts.requests
return reply, mts.errors[idx]
}
mts.headers, _ = metadata.FromIncomingContext(ctx)
mts.storage.AddSpans(exp)
return reply, nil
}
type mockCollector struct {
t *testing.T
traceSvc *mockTraceService
endpoint string
stopFunc func()
stopOnce sync.Once
}
type mockConfig struct {
errors []error
endpoint string
}
var _ collectortracepb.TraceServiceServer = (*mockTraceService)(nil)
var errAlreadyStopped = fmt.Errorf("already stopped")
func (mc *mockCollector) stop() error {
var err = errAlreadyStopped
mc.stopOnce.Do(func() {
err = nil
if mc.stopFunc != nil {
mc.stopFunc()
}
})
// Give it sometime to shutdown.
<-time.After(160 * time.Millisecond)
// Getting the lock ensures the traceSvc is done flushing.
mc.traceSvc.mu.Lock()
defer mc.traceSvc.mu.Unlock()
return err
}
func (mc *mockCollector) Stop() error {
return mc.stop()
}
func (mc *mockCollector) getSpans() []*tracepb.Span {
return mc.traceSvc.getSpans()
}
func (mc *mockCollector) getResourceSpans() []*tracepb.ResourceSpans {
return mc.traceSvc.getResourceSpans()
}
func (mc *mockCollector) GetResourceSpans() []*tracepb.ResourceSpans {
return mc.getResourceSpans()
}
func (mc *mockCollector) getHeaders() metadata.MD {
return mc.traceSvc.getHeaders()
}
// runMockCollector is a helper function to create a mock Collector
func runMockCollector(t *testing.T) *mockCollector {
return runMockCollectorAtEndpoint(t, "localhost:0")
}
func runMockCollectorAtEndpoint(t *testing.T, endpoint string) *mockCollector {
return runMockCollectorWithConfig(t, &mockConfig{endpoint: endpoint})
}
func runMockCollectorWithConfig(t *testing.T, mockConfig *mockConfig) *mockCollector {
ln, err := net.Listen("tcp", mockConfig.endpoint)
if err != nil {
t.Fatalf("Failed to get an endpoint: %v", err)
}
srv := grpc.NewServer()
mc := makeMockCollector(t, mockConfig)
collectortracepb.RegisterTraceServiceServer(srv, mc.traceSvc)
go func() {
Use the grpc.ClientConn to handle connections for the otlptracegrpc client (#2329) * POC using the grpc.ClientConn to handle connections * Update invalid client security test * Update client start test for a bad endpoint * Use any ClientConn a user provides * Connect ReconnectionPeriod to gRPC conn retries * Replace connection retry handling direct in otlptracegrpc * Fix client comments * Fix comment for NewGRPCConfig * Replace reconnection test * Fix grammar * Remove unrelated changes * Remove connection pkg * Rename evaluate to retryable * POC using the grpc.ClientConn to handle connections * Replace connection retry handling direct in otlptracegrpc * Add ClientConn use changes to changelog * Update otlptracegrpc options * Only close ClientConn that the Client create * Remove listener wrapper from mock_collector_test This is not needed now that no tests relies on the listener to wait for a connection to be established before continuing. * Fix spelling error * Do not use deprecated options in the otel-collector example * Add unit tests for retryable and throttleDelay funcs * Add unit tests for context heredity * Add test that exporter stop is linked to context cancel * go mod tidy * Update exporters/otlp/otlptrace/otlptracegrpc/client.go Co-authored-by: Anthony Mirabella <a9@aneurysm9.com> * Fix go.mod from rebase * Remove wrong comment about client stop closing gRPC conn * Fix shutdown test cleanup Do not check the second call to the client Stop. There is no guarantee it will not error in normal operation. * Make lint fixes * Fix flaky unit test Use the internals of the client to explicit cancel the context returned from exportContext. This gets around the bug where the select in Stop may randomly choose the non-context Done case and avoid returning an error (also failing to cancel the context). * Remove deprecation To configure the client/exporter with environment variables these options are used. There is no way to fully remove these options without removing support for configuration with environment variables. Leave that decision and strategy determination to a separate PR. * Fix grammatical error in comment Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>
2021-11-25 18:06:21 +02:00
_ = srv.Serve(ln)
}()
mc.endpoint = ln.Addr().String()
mc.stopFunc = srv.Stop
return mc
}