memory leak in libmbus Python bindings (0.0.8 and 0.9.0) when parsing XML #222
Description
Hello,
I have discovered a significant memory leak when using the libmbus Python bindings to query M-Bus meters and parse the resulting XML. The leak only appears when both querying and XML parsing are combined; neither querying alone nor XML parsing alone causes a leak.
Environment
OS: Ubuntu on ARM64 (Khadas VIM1)
Python version: 3.8.10
libmbus version: both 0.0.8 and 0.9.0 tested. With 0.9.0, the memory leak is even larger (over 1 MB lost in 127 blocks after a short run), and CPU usage is extremely high (58–87% vs 4% with 0.0.8).
Python mbus package: mbus 0.0.1 (Python bindings to libmbus from rSCADA, installed from PyPI)
XML parser: defusedxml.ElementTree (also tested with xml.etree.ElementTree and minidom with the same leaky results)
How to reproduce
Here is a minimal script that demonstrates the leak (replace the meter ID and device as needed):
import time
import tracemalloc
import gc
from mbus.MBus import MBus
from mbus.MBusLowLevel import MBUS_ADDRESS_NETWORK_LAYER
from defusedxml import ElementTree as SafeET
DEVICE = '/dev/ttyUSB1'
LIBPATH = '/root/pyapp/libmbus.so.1' # set your
METER_ID = '2101420924343307'
if len(METER_ID) == 8:
METER_ID += 'FFFFFFFF'
def query_and_parse():
mbus = MBus(device=DEVICE, libpath=LIBPATH)
mbus.connect()
mbus.select_secondary_address(METER_ID)
mbus.send_request_frame(MBUS_ADDRESS_NETWORK_LAYER)
reply = mbus.recv_frame()
xml = mbus.frame_data_xml(mbus.frame_data_parse(reply))
mbus.disconnect()
del mbus
root = SafeET.fromstring(xml)
val_nodes = root.findall('.//Value')
values = []
for node in val_nodes:
try:
values.append(float(node.text))
except Exception:
if node.text:
values.append(node.text)
root.clear()
del root
del xml
del values
def test_combined_leak(iterations=200, sleep_time=0.1):
tracemalloc.start()
for i in range(iterations):
try:
query_and_parse()
except Exception as e:
print(f"Exception at iteration {i}: {e}")
if i % 100 == 0:
current, peak = tracemalloc.get_traced_memory()
print(f"Iteration {i}: Current memory usage: {current / 1024:.1f} KB; Peak: {peak / 1024:.1f} KB")
time.sleep(sleep_time)
gc.collect()
tracemalloc.stop()
if name == "main":
test_combined_leak()
Valgrind Leak Summary
After running the above script under valgrind, I see the following leak summary:
==3593065== LEAK SUMMARY:
==3593065== definitely lost: 336,920 bytes in 41 blocks
==3593065== indirectly lost: 298,760 bytes in 679 blocks
==3593065== possibly lost: 20,216 bytes in 43 blocks
==3593065== still reachable: 720,421 bytes in 365 blocks
==3593065== suppressed: 0 bytes in 0 blocks
Additional Notes
The leak does not occur when only querying with libmbus (no XML parsing).
The leak does not occur when only parsing static XML (no libmbus).
The leak persists even with forced garbage collection.
The leak is visible in both Python and process memory, and confirmed by valgrind
(file attached for 0.9.0).
Please let me know if you need further details or if I can assist with testing any fixes or workarounds.