Segmentation Fault (Core Dumped)

[hashsiddique]

Hi! I'm Hassan Naeem. I'm facing an issue while executing ID2T code. The code gives "Segmentation Fault (Core Dumped)" at the output, once the pcap file is input. Can anyone help me out in this regard please? Why I'm getting such an error.

Screenshot of error attached please.

[hashsiddique]

@pepper-jk .. can you help me out please?

[pepper-jk]

Sorry, I did not get a notification until you mentioned me. I'll take a look at this now.
(also I'm fixing my notification settings.)

[pepper-jk]

Could you provide a pcap file that produces the error? So that I can reproduce this.
Are there pcap files that work for you? Or in other words does this happen with every pcap you have tried so far?

Also what OS / Linux distribution and version are you running?
Any issues during build?

Possibly information about your python setup:

• python version
• are all dependencies installed correctly?

Other notes:

• .pcap.ng files are not supported and those are sometimes saved with the .pcap extension. It might be possible that you got such a file. Try to convert the file to a normal pcap with a tool like wireshark and try again.
• do you have enough space left on your home partition / in your home directory to create the database?

[hashsiddique]

@pepper-jk Thank you so much for getting back, it means a lot as I'm stuck on this issue for some time now.

The discussed pcap file (datagusta.pcap) that is giving errors had pcapng extension, but I converted its extension to pcap through wireshark (save as option). Still it will not work?

Yes, I have tried other pcap files (02 more actually) and they work fine.

Ubuntu 20.04.2 LTS
Python Version 3.8.5
The final attempt of building the package and dependencies installation was successful. (Initially I was facing issues building up and installing dependencies).

Yes, I have enough space on Home partition (35 GBs).

[hashsiddique]

@pepper-jk I think there is some issue in recognizing pcap file converted (from pcapng) using wireshark and gives error (Segmentation Fault). Can you refer me any other converter (to convert pcapng to pcap) please (and ID2T program also recognizes that file)?

[pepper-jk]

The discussed pcap file (datagusta.pcap) that is giving errors had pcapng extension, but I converted its extension to pcap through wireshark (save as option). Still it will not work?

That sounds strange.

Are you sure, you changed the file format not just the extension in the file name?

[pepper-jk]

@pepper-jk I think there is some issue in recognizing pcap file converted (from pcapng) using wireshark and gives error (Segmentation Fault). Can you refer me any other converter (to convert pcapng to pcap) please (and ID2T program also recognizes that file)?

I have not worked with pcaps in a while, with a bit of googling these turned up:

• This free online converter by NETRESEC (Be careful! This is HTTP only, so if you have sensitive data, don't use this! Also if you don't trust them, don't use it.)
• tsahrk is able to do it as well tshark -F pcap -r {pcapng file} -w {pcap file} (but this should provide the same results as wireshark)
• Tcpdump version 4.1.1 and later should also handle pcapng files. So you should be able to convert them. tcpdump -r {pcapng file} -w {pcap file}

Hope this helps, but in the future you should just search for tools like this yourself. As this is not directly related to the ID2T project.

[pepper-jk]

If you could upload the pcapng and pcap file here, I could try to reproduce the issue myself.

[LuccaRoofthooft]

Hello @pepper-jk
Any update on this? I am facing the same issue as Hassan Naeem. I can create attacks for some files but for some not. The files are all in pcap form, I tried to convert them again to pcap to be sure but it still gives the same error.

System set up information:
MATE 1.26.0 VM
Python version 3.10.6
130GB disk space left

[LuccaRoofthooft]

$ ./build.sh --full ouput

Updating submodules

Detected OS: Debian
Packages: Checking...
Packages: Found.
Requirement already satisfied: pip in ./.venv/lib/python3.10/site-packages (23.1)
Requirement already satisfied: cairocffi==0.8.1 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 1)) (0.8.1)
Requirement already satisfied: cffi>=1.12.2 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 2)) (1.15.1)
Requirement already satisfied: coverage>=4.5.1 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 3)) (7.2.3)
Requirement already satisfied: cycler>=0.10.0 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 4)) (0.11.0)
Requirement already satisfied: kiwisolver>=1.0.1 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 5)) (1.4.4)
Requirement already satisfied: lea==2.3.5 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 6)) (2.3.5)
Requirement already satisfied: matplotlib>=2.2.3 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 7)) (3.7.1)
Requirement already satisfied: memory-profiler>=0.54.0 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 8)) (0.61.0)
Requirement already satisfied: numpy>=1.15 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 9)) (1.24.2)
Requirement already satisfied: psutil>=5.4.5 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 10)) (5.9.4)
Requirement already satisfied: pycparser>=2.18 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 11)) (2.21)
Requirement already satisfied: pyparsing>=2.2.0 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 12)) (3.0.9)
Requirement already satisfied: python-dateutil>=2.7.2 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 13)) (2.8.2)
Requirement already satisfied: pytz>=2018.4 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 14)) (2023.3)
Requirement already satisfied: pyxdg>=0.26 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 15)) (0.28)
Requirement already satisfied: scapy==2.4.2 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 16)) (2.4.2)
Requirement already satisfied: scipy>=1.1.0 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 17)) (1.10.1)
Requirement already satisfied: six>=1.11.0 in ./.venv/lib/python3.10/site-packages (from -r resources/requirements.txt (line 18)) (1.16.0)
Requirement already satisfied: contourpy>=1.0.1 in ./.venv/lib/python3.10/site-packages (from matplotlib>=2.2.3->-r resources/requirements.txt (line 7)) (1.0.7)
Requirement already satisfied: fonttools>=4.22.0 in ./.venv/lib/python3.10/site-packages (from matplotlib>=2.2.3->-r resources/requirements.txt (line 7)) (4.39.3)
Requirement already satisfied: packaging>=20.0 in ./.venv/lib/python3.10/site-packages (from matplotlib>=2.2.3->-r resources/requirements.txt (line 7)) (23.1)
Requirement already satisfied: pillow>=6.2.0 in ./.venv/lib/python3.10/site-packages (from matplotlib>=2.2.3->-r resources/requirements.txt (line 7)) (9.5.0)
-- Running Release configuration.
-- CMake version: 3.22.1
-- CMAKE_CXX_COMPILER '/usr/bin/c++' 'GNU' '11.3.0'
-- CMAKE_CXX_FLAGS                ''
-- CMAKE_CXX_FLAGS_RELEASE        '-O3 -DNDEBUG -Wall'
-- SQLITECPP_RUN_CPPLINT OFF
-- Could NOT find cppcheck
-- SQLITECPP_RUN_DOXYGEN OFF
-- SQLITECPP_BUILD_EXAMPLES OFF
-- SQLITECPP_BUILD_TESTS OFF
-- Tins library found in /usr/local/lib/libtins.so
-- Python includes found in: /usr/include/python3.10
-- Python libs found in: /usr/lib/x86_64-linux-gnu/libpython3.10.so
-- Configuring done
-- Generating done
-- Build files have been written to: /home/eaton/ID2T/code_boost/src/build
Consolidate compiler generated dependencies of target cpputils
Consolidate compiler generated dependencies of target SQLiteCpp
Consolidate compiler generated dependencies of target sqlite3
Consolidate compiler generated dependencies of target botnetcomm
[ 11%] Built target sqlite3
[ 22%] Building CXX object CMakeFiles/botnetcomm.dir/cxx/botnet_comm_processor.cpp.o
[ 55%] Built target SQLiteCpp
[ 61%] Building CXX object CMakeFiles/cpputils.dir/cxx/utilities.cpp.o
Consolidate compiler generated dependencies of target pcapreader
[ 72%] Building CXX object CMakeFiles/pcapreader.dir/cxx/statistics.cpp.o
[ 72%] Building CXX object CMakeFiles/pcapreader.dir/cxx/pcap_processor.cpp.o
[ 77%] Building CXX object CMakeFiles/pcapreader.dir/cxx/statistics_db.cpp.o
[ 83%] Building CXX object CMakeFiles/pcapreader.dir/cxx/utilities.cpp.o
In file included from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:16,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/botnet_comm_processor.h:10,
                 from /home/eaton/ID2T/code_boost/src/cxx/botnet_comm_processor.cpp:1:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h:194:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations]
  194 |         PyEval_InitThreads();
      |         ~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/python3.10/Python.h:130,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/common.h:112,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pytypes.h:12,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/botnet_comm_processor.h:10,
                 from /home/eaton/ID2T/code_boost/src/cxx/botnet_comm_processor.cpp:1:
/usr/include/python3.10/ceval.h:122:37: note: declared here
  122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~
In file included from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:16,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/utilities.cpp:2:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h:194:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations]
  194 |         PyEval_InitThreads();
      |         ~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/python3.10/Python.h:130,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/common.h:112,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pytypes.h:12,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/utilities.cpp:2:
/usr/include/python3.10/ceval.h:122:37: note: declared here
  122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~
In file included from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:16,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/utilities.cpp:2:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h:194:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations]
  194 |         PyEval_InitThreads();
      |         ~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/python3.10/Python.h:130,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/common.h:112,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pytypes.h:12,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/utilities.cpp:2:
/usr/include/python3.10/ceval.h:122:37: note: declared here
  122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~
In file included from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:16,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/pcap_processor.h:12,
                 from /home/eaton/ID2T/code_boost/src/cxx/pcap_processor.cpp:1:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h:194:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations]
  194 |         PyEval_InitThreads();
      |         ~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/python3.10/Python.h:130,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/common.h:112,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pytypes.h:12,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/pcap_processor.h:12,
                 from /home/eaton/ID2T/code_boost/src/cxx/pcap_processor.cpp:1:
/usr/include/python3.10/ceval.h:122:37: note: declared here
  122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~
In file included from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:16,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics_db.h:13,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics.cpp:8:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h:194:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations]
  194 |         PyEval_InitThreads();
      |         ~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/python3.10/Python.h:130,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/common.h:112,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pytypes.h:12,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics_db.h:13,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics.cpp:8:
/usr/include/python3.10/ceval.h:122:37: note: declared here
  122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~
In file included from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:16,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics_db.h:13,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics_db.cpp:1:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h: In function ‘pybind11::detail::internals& pybind11::detail::get_internals()’:
/home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/internals.h:194:27: warning: ‘void PyEval_InitThreads()’ is deprecated [-Wdeprecated-declarations]
  194 |         PyEval_InitThreads();
      |         ~~~~~~~~~~~~~~~~~~^~
In file included from /usr/include/python3.10/Python.h:130,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/detail/common.h:112,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pytypes.h:12,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/cast.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/attr.h:13,
                 from /home/eaton/ID2T/code_boost/src/pybind11/include/pybind11/pybind11.h:43,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics_db.h:13,
                 from /home/eaton/ID2T/code_boost/src/cxx/statistics_db.cpp:1:
/usr/include/python3.10/ceval.h:122:37: note: declared here
  122 | Py_DEPRECATED(3.9) PyAPI_FUNC(void) PyEval_InitThreads(void);
      |                                     ^~~~~~~~~~~~~~~~~~
[ 88%] Linking CXX shared library libcpputils.so
[ 88%] Built target cpputils
[ 94%] Linking CXX shared library libbotnetcomm.so
[ 94%] Built target botnetcomm
[100%] Linking CXX shared library libpcapreader.so
[100%] Built target pcapreader


All is set. ID2T is ready.

Run efficiency tests with the command './test_efficiency'
Run unit tests with the command './run_tests'
Run ID2T with the command './id2t'

[pepper-jk]

@LuccaRoofthooft there is no update on this from me. I'm not sure if @hashsiddique solved the problem themself or if they just lost interest. Without more information I was not able to help.

Your log shows some interesting warnings regarding pybind methods deprecated at or after python 3.9. Maybe this is the reason, you have issues. So I would suggest setting up a python 3.8 and 3.9 environment and see if you can reproduce both the warnings from the build script and your actual problem.

Unfortunately, I currently have little time to investigate issues for ID2T. But I still suggest opening a separate issue, as we can not be sure yours is related to this one. Especially as you both use different python versions. In your issue please elaborate on your actual issue. Are you also getting a Segmentation Fault (Core Dumped)? Could you provide a pcap file that works and one that does not? Also please link to this comment from the new issue. Thank you very much.