Added x64 function prologue

[N0fix]

Adding x64 function prologues:

55       push rbp
48 89 e5 mov rbp, rsp

55       push rbp
48 89 ec mov rsp, rbp

I found the first one 55 48 89 E5 in a CTF executable.
See the executable attached. SMDA is missing the entry point, figured with some debug that it was not finding the prologue. It also avoid the function at EP because it ends with a syscall(exit), and tries to continue disassembling after that. But for now, adding the prologue will do.
I assumed the second opcode 55 48 89 EC) should exist too.

[r0ny123]

Hi @N0fix,

Thanks for the PR! I've investigated the test failure in tests/testFileFormatParsers.py and have some good news.

The failure AssertionError: 211 != 208 in testMacOsParsingWithKomplex is actually a positive sign. It indicates that your addition of the standard x64 prologue 55 48 89 e5 (push rbp; mov rbp, rsp) has improved SMDA's coverage. Specifically, it now correctly identifies a valid function at 0x100006c80 that was previously missed. Finding this function also allowed the analyzer to bridge a gap and identify two additional code blocks/functions (0x1000017a8 and 0x100006d84), bringing the total count to 211.

However, regarding the second prologue you added (55 48 89 ec / push rbp; mov rsp, rbp): This sequence is quite unusual for a function start and looks more like a stack restoration or epilogue sequence. My analysis showed it wasn't found in the komplex binary, so it didn't contribute to the improved results. To avoid potential false positives in other binaries, I suggest we drop this specific prologue for now.

Could you please update your PR with the following changes?

1. Remove b"\x55\x48\x89\xec" from DEFAULT_PROLOGUES in smda/intel/definitions.py.
2. Update tests/testFileFormatParsers.py line 127 (in testMacOsParsingWithKomplex):

self.assertEqual(komplex_unmapped_disassembly.num_functions, 211)

Thanks again for the improvement!

[N0fix]

Hey,

Thanks for the extensive testing. I assumed that push rbp; mov rsp, rbp could happen in some prologues, but I didn't took the time to check.
I removed this prologue and updated the test, as you asked.

Have a good day

[danielplohmann]

Hey!
first, thanks for the pull request!
Actually, I'd argue that it would be okay to add both variants, but I noticed that there was possibly a minor mistake.
Adding b"\x55\x48\x8b\xec" instead of b"\x55\x48\x89\xec" would also yield push rbp; mov rbp, rsp as this would be the correct variant with flipped direction bit in the encoded instruction. compare that with the 32bit variants already present.

Now, before adding the prologues, I'd like to benchmark this in a wider scenario, for which I'm going to use the ground truth created in my PhD thesis (probably using the patched ByteWeight and the annotated Malpedia excerpt, which however has only a few 64bit binaries).
Expect me to come back with some results in the near future, to get you some better insights into how this improved function coverage. ;)

[danielplohmann]

Alright, I took a detour looking at your provided executable and noticed that the OEP function was also not detected at all.
I have now addressed issues with handling exit() syscalls and also removed an erroneous return in the ELF symbol parser, which should improve overall coverage for ELF binaries.

I only benchmarked it for now against a set of Windows executables and didn't notice any performance degradation (but an ever so slight improvement in precision for the benign x64).

Thanks for the contribution!