When debugging this crash, we noticed that the crash reporter pointed to a memory allocation error, which indicated that the decoder was attempting to allocate memory at a location where an object already existed. The decoder’s GetBitString method was attempting to allocate memory at a specific index, which was out of range for the type that was being allocated. This is the exact scenario where an index out of range panic occurs. When the AMF and NGAP decoders detect that the memory being pointed to by the bit string is already allocated, they will panic. With a bit of effort and a keen eye, we were able to locate the error in the NGAP NGAP decoder. At the time of this writing, NGAP 3.2.1 has been released, and the NGAP decoder has been patched.
Crash Reporting
& Memory Allocation
Memory allocation errors are a common occurrence in all programming languages and can be quickly fixed, if attention is paid to the specifics of the error. For example, when an index out of range panic occurs, it could be due to an attempt to allocate memory at a specific index that was out of range for the type that was being allocated. If this situation occurred often enough, it would eventually lead to crashes. In these cases, a compiler would need to be updated with information about what types were allocated where. This re-compilation process can now be automated by using memory debugging tools like Valgrind and OProfile. However, there are cases where memory allocation errors are not as easy to detect or fix because they happen infrequently or they occur deep within a code base in which recompiling is not possible. These situations require more work on the developer’s part when debugging memory leaks.
Debugging an Encoding Crash
The NGAP decoder was a bit more difficult to track down, but it also does not require any code changes. The error that we encountered in the NGAP decoder can be seen in the function GetBitString in the file ngaa-decoder.c. There is one instance of the following code:
if (stride > 0) {
A stride is an index in an array, and this particular location will always point to a non-existent object, so a stride out of range panic will occur when attempting to allocate memory for this object at this index. If there are multiple instances of this code that point to this location, then each instance will get its own allocation panic when the object does not exist. We originally suspected that there may have been multiple allocation panics being triggered by different conditions in different places, but it turned out that all occurrences were exactly like this one, which was easy to debug. Another method of debugging these crashes is through monitoring system logs while encoding video streams or decoding streams and watching for specific errors.
Crash in decoder when bit string is not sufficiently aligned
The decoder will attempt to allocate memory for a bit string at a specific index. The index is passed in as an offset, and the allocation can fail if it is not sufficiently aligned with the pointer being pointed to. In this particular case, the compiler was attempting to allocate memory that would be past the end of the allocated block of memory.
Timeline
Published on: 10/24/2022 14:15:00 UTC
Last modified on: 10/25/2022 14:04:00 UTC