TensorFlow is one of the most popular open-source platforms for machine learning. Recently, a serious vulnerability has been found in how TensorFlow handles certain special types of data called CompositeTensorVariant tensors. This bug is tracked as CVE-2022-41909, and in this post, I’ll break down what the issue is, show what could go wrong, and explain how the TensorFlow team fixed it.
What’s the Problem?
At the heart of TensorFlow’s problem is the function tf.raw_ops.CompositeTensorVariantToComponents. This function is supposed to safely handle a special TensorFlow object called a CompositeTensorVariant. However, researchers found that if you feed this function an invalid or malformed tensor, TensorFlow will crash with a segmentation fault (“segfault”). A segfault is when a program tries to access an area of memory it shouldn't—which can result in denial of service or, in worse cases, even open a path for attackers to exploit the system.
TensorFlow 2.10. up to 2.10.1
TensorFlow 2.11 will include the fix and older versions will have the fix backported (“cherry-picked”).
Vulnerability Details
Inside the function, there was no check for whether the encoded input is actually a valid CompositeTensorVariant. If a bad input is given, the function would try to treat it as valid and crash the entire process.
Vulnerable example
import tensorflow as tf
# This is NOT a valid CompositeTensorVariant tensor.
bad_input = tf.constant([1,2,3])
# This will crash Python!
components = tf.raw_ops.CompositeTensorVariantToComponents(
encoded=bad_input,
component_types=[tf.float32])
print(components)
If you run this code on an affected version of TensorFlow, your process will crash. On some systems, this could enable a denial of service—imagine a production server crashing just because a user sent a bad input!
To show how this bug works, here’s a simple script you can try (on a test system only!)
import tensorflow as tf
# Deliberately pass wrong tensor type to the function
try:
tf.raw_ops.CompositeTensorVariantToComponents(
encoded=tf.constant(123),
component_types=[tf.float32]
)
except Exception as e:
print("Caught Exception:", e)
On patched versions, you’ll get an error like
InvalidArgumentError: input is not a valid CompositeTensorVariant tensor
But on vulnerable versions, this code would crash Python with a segmentation fault.
How Did They Patch It?
The fix is surprisingly simple: add input validation to ensure the function only works with valid CompositeTensorVariant tensors. If the input isn’t valid, the function now correctly throws an exception instead of crashing.
You can see the exact changes in these commits
- bf594d08d377dc6a3354d9fdb494b32d45f91971
- 660ce5a89eb6766834bdc303d2ab3902aef99d3d
Here’s a snippet from the real patch
if (!IsCompositeTensorVariant(encoded)) {
context->CtxFailure(errors::InvalidArgument("input is not a valid CompositeTensorVariant tensor"));
return;
}
What Should You Do?
- Update TensorFlow! The fix will be included in TensorFlow 2.11+. If you are on TensorFlow 2.10.1, 2.9.3, or 2.8.4, download the latest patch as soon as it’s available.
- Validate Inputs: Always sanitize and validate inputs to TensorFlow models, especially if you're building a web API or service that lets users upload data.
- Monitor for Segfaults: Unusual crashes or segfaults in your TensorFlow code could be a sign of something malicious or unexpected; investigate and patch immediately.
Original References
- GitHub Security Advisory GHSA-h2j4-qw2w-qfr9
- TensorFlow Commit bf594d08
- TensorFlow Commit 660ce5a8
- CVE-2022-41909 NIST page
Final Thoughts
CVE-2022-41909 reminds us that even mature libraries like TensorFlow can have nasty bugs, especially in rarely used corners of their APIs. If you use TensorFlow in production, update your dependencies regularly and scan the change logs for security issues. And if you’re a developer, check user-supplied data everywhere—it could save you (and your users) from a bad day.
Timeline
Published on: 11/18/2022 22:15:00 UTC
Last modified on: 07/10/2023 16:50:00 UTC