Mem stomps are the worst. At least nowadays we have address sanitizer to find em. We recently had one where the same 4 byte pattern was being written randomly in different places in memory, would happen all over the place. Always the same 4 byte pattern, just different places. Eventually, it would write to a spot that was being used and cause a crash. Different callstacks almost every time, but the same memory footprint wherever the crash happened. An array size, a memory address, a string mangled, etc. Eventually we got our ASan build working after about a month of trying to track it down, digging through callstacks and core dumps. We found that it was a dangling pointer in our AI system, when an AI was removed, there was a situation where the pointer wouldn’t always be cleaned up, then later when another AI was removed, a boolean and an enum were written to the address of the dangling pointer, always the same format/value. which had haunted us for so long.
Mem stomps are the worst. At least nowadays we have address sanitizer to find em. We recently had one where the same 4 byte pattern was being written randomly in different places in memory, would happen all over the place. Always the same 4 byte pattern, just different places. Eventually, it would write to a spot that was being used and cause a crash. Different callstacks almost every time, but the same memory footprint wherever the crash happened. An array size, a memory address, a string mangled, etc. Eventually we got our ASan build working after about a month of trying to track it down, digging through callstacks and core dumps. We found that it was a dangling pointer in our AI system, when an AI was removed, there was a situation where the pointer wouldn’t always be cleaned up, then later when another AI was removed, a boolean and an enum were written to the address of the dangling pointer, always the same format/value. which had haunted us for so long.