Scudo backtrace in libc.so while running automation in Android13 environment

[Yavnikaa]

Description:
While migrating unit tests from Android 10 to Android 13 some of the UTs are crashing during execution with a Scudo stacktrace pointing to libc. I’ve attached the relevant logcat output for reference.

05-09 15:30:59.904  4905  4905 F DEBUG   : signal 6 (SIGABRT), code -1 (SI_QUEUE), fault addr --------
05-09 15:30:59.904  4905  4905 F DEBUG   : Abort message: 'Scudo ERROR: corrupted chunk header at address 0x74fc4d862880'
05-09 15:30:59.905  4905  4905 F DEBUG   :     rax 0000000000000000  rbx 00000000000012cc  rcx 000074feca8c070f  rdx 0000000000000006
05-09 15:30:59.905  4905  4905 F DEBUG   :     r8  0000000000000000  r9  0000000000000000  r10 000074fbb2512a00  r11 0000000000000213
05-09 15:30:59.905  4905  4905 F DEBUG   :     r12 0000000000000000  r13 000074fbb2513050  r14 000074fbb25129f8  r15 0000000000001326
05-09 15:30:59.905  4905  4905 F DEBUG   :     rdi 00000000000012cc  rsi 0000000000001326
05-09 15:30:59.906  4905  4905 F DEBUG   :     rbp 0000000000000000  rsp 000074fbb25129f0  rip 000074feca8c070f
05-09 15:30:59.906  4905  4905 F DEBUG   : backtrace:
05-09 15:30:59.906  4905  4905 F DEBUG   :       #00 pc 000000000005f70f  /apex/com.android.runtime/lib64/bionic/libc.so (abort+191) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.906  4905  4905 F DEBUG   :       #01 pc 0000000000048de5  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::die()+5) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.907  4905  4905 F DEBUG   :       #02 pc 00000000000493d8  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::ScopedErrorReport::~ScopedErrorReport()+24) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.907  4905  4905 F DEBUG   :       #03 pc 00000000000494c2  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::reportHeaderCorruption(void*)+66) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.907  4905  4905 F DEBUG   :       #04 pc 000000000004ace9  /apex/com.android.runtime/lib64/bionic/libc.so (scudo::Allocator<scudo::AndroidConfig, &(scudo_malloc_postinit)>::deallocate(void*, scudo::Chunk::Origin, unsigned long, unsigned long)+313) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.908  4905  4905 F DEBUG   :       #05 pc 00000000000cd646  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_key_clean_all()+134) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.908  4905  4905 F DEBUG   :       #06 pc 00000000000ccecd  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_exit+77) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.909  4905  4905 F DEBUG   :       #07 pc 00000000000ccda2  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+66) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.909  4905  4905 F DEBUG   :       #08 pc 0000000000060d47  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+55) (BuildId: f090904cc3ac285a6f190f8003c3eb0e)
05-09 15:30:59.933   211   211 E tombstoned: Tombstone written to: tombstone_07
05-09 15:30:59.963   534   640 I BootReceiver: Copying /data/tombstones/tombstone_07 to DropBox (SYSTEM_TOMBSTONE)
05-09 15:30:59.965   534   640 I DropBoxManagerService: add tag=SYSTEM_TOMBSTONE isTagEnabled=true flags=0x6
05-09 15:30:59.998   534  4911 I DropBoxManagerService: add tag=data_app_native_crash isTagEnabled=true flags=0x2
05-09 15:31:00.020   534  4910 W ActivityTaskManager:   Force finishing activity com.microsoft.motif.AndroidTestHost/.MainActivity
05-09 15:31:00.034   534   640 I DropBoxManagerService: add tag=SYSTEM_TOMBSTONE_PROTO isTagEnabled=true flags=0x4
05-09 15:30:59.994     0     0 I init    : Untracked pid 4905 exited with status 0
05-09 15:31:00.002     0     0 I init    : Untracked pid 4905 did not have an associated service entry and will not be reaped
05-09 15:31:00.024     0     0 I init    : Untracked pid 4907 exited with status 0
05-09 15:31:00.033     0     0 I init    : Untracked pid 4907 did not have an associated service entry and will not be reaped
05-09 15:31:00.046     0     0 I binder  : undelivered transaction 257865, process died.
05-09 15:31:00.094   310   310 I Zygote  : Process 4812 exited due to signal 6 (Aborted)

This crash is currently blocking further progress, as we don't have unstripped libc.so which is why we can't deobfuscate further, nor figure out the root cause. Is this a known issue? Can you help us proceed here?

[DanAlbert]

corrupted chunk header at address 0x74fc4d862880

This means memory corruption, which is almost certainly an app bug (whether your code or from one of your dependencies). The error isn't scudo's, scudo is reporting the error. @cferris1000, if that can be reworded, that might help.

This doc explains the tools available to help you track down the source of the memory corruption: https://developer.android.com/ndk/guides/memory-debug.

[enh-google]

corrupted chunk header at address 0x74fc4d862880

This means memory corruption, which is almost certainly an app bug (whether your code or from one of your dependencies). The error isn't scudo's, scudo is reporting the error. @cferris1000, if that can be reworded, that might help.

yeah, there's a reason why i changed the ambiguous "jni error" from dalvik/art to explicitly say "app bug"...

[Yavnikaa]

Thanks! We did try to use MTE before but since our test app currently runs on Android 13, we couldn't really test out with MTE.
Yes, the address is here but is there any way we can know the corresponding so? I tried to resolve the address through llvm-addr2line with our available in-house built unstripped SOs but didn't get the desired output and therefore assumed that it could be a known existing issue. Given that it is an app bug, how can I debug it further starting from mapping corrupted address to the relevant SO?

[DanAlbert]

addr2line won't help because you can't corrupt text sections. Those are not writable. This is heap corruption, and the way to correlate the corrupting write with the culprit is to use one of those tools in my other post. HWASan is probably your best bet.