Add NTDS parser

[joost-j]

Adding a parser for the New Technology Directory Services Directory Information Tree (NTDS.dit) file, present on Windows Domain Controllers. This type of ESE database holds information related to an Active Directory environment.

Although multiple people have contributed to this tool within Fox-IT over the years, special acknowledgement goes to colleagues @Schamper and Aman Asarfi, whose work and research were invaluable. Performing in-depth investigation of the file format mainly comes from their end, and this pull request mainly puts all the code and knowledge gathered over the years into a proper Dissect-compatible format.

Several features include:

• Interacting with the NTDS database using Python functions (e.g. calling ntds.users()
• Interacting with the NTDS via LDAP queries (e.g. being able to call ntds.query("(objectClass=user)")
• Optimizations and caching to speed up the database lookups
• Testing most of the functionality above

Still work in progress:

• Create a wrapper tool around the NTDS class that outputs data in Bloodhound-supported format. Within Fox this goes by the namefoxhound. This tool can be added later on as well in a separate pull request.
• Add more objects and useful functions. E.g. a Domain object, and more functionality related to checking whether objects are related to each other. For example, a function where you could callObject.has_priv_over(obj2, GENERIC_ALL).
• Increase test coverage
• Adjusting NTDS.dit Plugin dissect.target#1347 or this code to be compatible
• Further code readability & style improvements

Related to fox-it/dissect.target#1348

[codecov]

Codecov Report

❌ Patch coverage is 0% with 1588 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (1e58b09) to head (3e0aa64).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
dissect/database/ese/ntds/database.py	0.00%	178 Missing ⚠️
dissect/database/ese/ntds/util.py	0.00%	148 Missing ⚠️
dissect/database/ese/ntds/objects/object.py	0.00%	139 Missing ⚠️
dissect/database/ese/ntds/schema.py	0.00%	115 Missing ⚠️
dissect/database/ese/ntds/objects/__init__.py	0.00%	105 Missing ⚠️
dissect/database/ese/ntds/sd.py	0.00%	101 Missing ⚠️
dissect/database/ese/ntds/query.py	0.00%	86 Missing ⚠️
dissect/database/ese/ntds/pek.py	0.00%	67 Missing ⚠️
dissect/database/ese/cursor.py	0.00%	41 Missing ⚠️
dissect/database/ese/ntds/ntds.py	0.00%	29 Missing ⚠️
... and 112 more

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #8       +/-   ##
==========================================
- Coverage   82.05%   0.00%   -82.06%     
==========================================
  Files          31     146      +115     
  Lines        2352    3881     +1529     
==========================================
- Hits         1930       0     -1930     
- Misses        422    3881     +3459     

Flag	Coverage Δ
unittests	0.00% <0.00%> (-82.06%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Schamper]

Okay I think I've fueled my addiction enough for now. In my opinion we only need to add a few more unit tests based on GOAD (I already added a test file) and perhaps replace the existing tests with the GOAD file. Most of everything "basic" (and a little more) is already done. We can work on improving the "useful objects" later, as well as the additional tooling.

The only part I'm a little divided on are the bajillion source files for all objects, but I guess there's not really a nicer way to do that.

[joost-j]

Huge thanks to @Schamper for improving, extending and refactoring the code a lot; you went way deeper into the cruel depths of NTDS.dit than I originally did 😄. I think that as of now, the code is very much in a version 1.0 state and ready to be reviewed.

[codspeed-hq]

CodSpeed Performance Report

Congrats! CodSpeed is installed 🎉

🆕 6 new benchmarks were detected.

You will start to see performance impacts in the reports once the benchmarks are run from your default branch.

Detected benchmarks

• test_benchmark_groups[goad]: 19.5 ms
• test_benchmark_users[goad]: 8 ms
• test_benchmark_users[large]: 10.6 s
• test_benchmark_computers[goad]: 1.3 ms
• test_benchmark_groups[large]: 295.8 ms
• test_benchmark_computers[large]: 3.7 s

[Horofic]

🗣️🔔

[Horofic]

Please update return types of the root_domain and pek in ntds.py. Will ack afterwards.