In the world of cloud-native infrastructure, HashiCorp Nomad is a powerful tool for deploying and managing applications. But even the best tools can have critical vulnerabilities. In this long-read, we’ll dive into the details of CVE-2022-3867—a security flaw in Nomad and Nomad Enterprise that could’ve allowed token-based event stream subscribers to snoop on updates long after their access should have expired.
We’ll break down what went wrong, see some code that explains the bug, and give you everything you need to know to stay secure.
What is HashiCorp Nomad?
If you’re not already familiar, HashiCorp Nomad is a flexible workload orchestrator—think of it as a tool to automatically launch and manage applications (like websites or batch jobs) across many servers. Security is key for these tools because they often manage sensitive workloads.
Severity: Moderate (according to HashiCorp)
What’s the problem?
Event stream subscribers in Nomad authenticate using tokens (think “hall passes” to prove they can see certain events). These tokens can have a time-to-live (TTL), after which they should expire and no longer grant access. But in these versions, even after a token’s TTL was up, an event stream subscriber kept receiving updates until the server actually ran its garbage collection routine—a process that might take much longer than expected.
So, a revoked or naturally-expired token could keep “listening in” on sensitive system or job changes for longer than it should.
How Did This Happen?
Nomad uses an event streaming interface (think: a firehose of updates about jobs, allocations, node health, etc.). To listen, clients must authenticate with a token. Tokens have a TTL set for safety. When TTL is up, the client’s access should stop _immediately_. That wasn’t happening!
Why not?
The internal code kept the event subscriber alive as long as the subscription existed—even if the token had expired. Actual cleanup only happened when the server next ran “garbage collection” for expired tokens, which could be delayed.
Alice’s token times out.
- But the backend still lets her subscription receive new event data! As long as server garbage collection hasn't kicked in, her “expired” token is still good.
Here’s a simplified mock of what went wrong in Go (the language Nomad uses)
type Subscriber struct {
token *Token
lastEvent int
}
func (s *Subscriber) receiveEvent(event Event) {
if s.token.IsExpired() {
// Should remove subscriber!
return
}
s.send(event)
}
func (server *EventServer) sendToSubscribers(event Event) {
for _, sub := range server.subscribers {
// Oops! No check for token expiration here!
sub.receiveEvent(event)
}
}
In reality, the cleanup was only happening periodically (when garbage collection ran), so expired tokens could “live on” way longer than expected.
Why is this a Big Deal?
Nomad event streams can carry confidential information about jobs, resource usage, or security events. If a user’s token (or, worse, a compromised token) is supposed to have expired, the system should immediately cut off access.
Attacker steps
1. Attacker obtains a TTL-limited Nomad token with event stream privileges (legitimately or via compromise).
Bug: Event updates keep flowing!
5. Attacker listens for minutes or more, even after their token is dead—until garbage collection runs and finally revokes the subscription.
> Note: This requires the attacker to already have a token, but is a risk in shared/delegated environments.
How Was It Fixed?
In Nomad 1.4.2, developers ensured that event stream subscribers are _immediately_ disconnected when their tokens expire. Subscriptions tied to an expired token are now promptly terminated, not left alive until garbage collection.
Official HashiCorp Advisory:
- Nomad Security Advisory (CVE-2022-3867)
- GitHub Issue
Remediation Steps
1. Upgrade to 1.4.2 immediately (download).
Conclusion
CVE-2022-3867 serves as a classic example of why token management and timely revocation are so critical in cloud-native platforms like Nomad. Even a small bug in garbage collection can accidentally give extra “hang time” to users or tokens whose access should be over.
Remember: Always keep your infrastructure updated – and double-check how TTLs are actually enforced!
Further Reading
- Official HashiCorp CVE-2022-3867 Advisory
- HashiCorp Nomad Releases
- Understanding Nomad's Event Stream
- Nomad Security Best Practices
Timeline
Published on: 11/10/2022 06:15:00 UTC
Last modified on: 11/15/2022 17:24:00 UTC