Major Refactor for MicroPython Efficiency (v2.0.0)

[tadeubas]

This PR introduces breaking changes and should be released as v2.0.0.

Benchmarks are detailed below. This achieves: firmware size reduction, less RAM usage and fast code execution!

This PR removes ~600 lines of code (without new lines from tests and poetry.lock)

Summary of Changes

• Split bytewords.py into two focused modules, following the library’s usage pattern (encode and decode are rarely needed together):

  • bytewords_encode.py
  • bytewords_decode.py

• Unified decoder architecture:

  • Refactored ur_decoder.py and fountain_decoder.py to share a common base class, BasicDecoder.

• Simplified CRC handling:

  • Removed the custom crc32 implementation in crc32.py in favor of binascii.crc32, which is already available in Krux.

• Code cleanup and simplification:

  • Removed unnecessary checks, unused constants/functions, custom error classes, and leftover debug/test code.

• MicroPython optimizations:

  • Refactored code to reduce RAM usage and improve CPU efficiency.
  • Encoded data is shown as UPPERCASE as default (lower QR code size)

• Testing and documentation:

  • Added new tests and increased overall test coverage.
  • Updated README.md with revised usage and setup instructions.

• Tooling additions:

  • Added development dependencies:

    • pytest
    • pytest-cov
    • pylint
    • vulture
    • black

• Configuration updates:

  • Added .pylintrc
  • Added poetry.lock

[tadeubas]

Benchmark Results (MicroPython)

The firmware size reduction with v2.0.0 was ~10kB.

The benchmark uses arbitrary input data (81 bytes) multiplied by a scale factor to stress execution time and memory usage.

In v0.1.0, the garbage collector is already triggered at a scale factor of 3, so the comparison focuses primarily on execution time. Memory measurements are still included for reference, and the full raw data is provided below.

Metric (scale factor)	v0.1.0	v2.0.0	Improvement
time diff (1)	632	64	~10x faster
memory diff (1)	583232	116640	~5x less
time diff (10)	5165	352	~15x faster
time diff (20)	11720	947	~12x faster
time diff (100)	69358	9639	~7x faster
time diff (200)	176719	29967	~6x faster

Benchmark Setup

• MICROPY_ENABLE_COMPILER was set to 1 in:
  firmware/MaixPy/components/micropython/port/include/mpconfigport.h
• The benchmark was executed via the MaixPy IDE terminal.
• Portions of the code can be uncommented to validate correctness during execution.

from krux.wdt import wdt

wdt.stop()

import time
import gc
gc.collect()

print("\n\n------ start foundation-ur-py test\n\n")

# memory control
mem_list = []
mem_list.append(gc.mem_free())

from ur.cbor_lite import CBOREncoder, CBORDecoder
from ur.ur_decoder import URDecoder
from ur.ur_encoder import UREncoder
from ur.ur import UR

scale_factor = 1
data = b'1234567890abcdefghijklmnopqrstuvwxyz-=+_/.,;:ABCDEFGHIJKLMNOPQRSTUVWXYZ0987654321'
data = data * scale_factor

# time control
time_b = time.ticks_ms()

# Encode CBOR
cbor_enc = CBOREncoder()
cbor_enc.encodeBytes(data)

# memory control
mem_list.append(gc.mem_free())

encoded = cbor_enc.get_bytes()
#print("CBOREncoder")
#print(encoded)

# memory control
mem_list.append(gc.mem_free())

## Decode CBOR
cbor_dec = CBORDecoder(encoded)

# memory control
mem_list.append(gc.mem_free())

decoded, _ = cbor_dec.decodeBytes()

# memory control
mem_list.append(gc.mem_free())

decoded = bytes(decoded)
#print("\nCBORDecoder")
#print(decoded)

#assert(data == decoded)

# memory control
mem_list.append(gc.mem_free())

# ----------

# UR
ur_obj = UR("bytes", encoded)
#print("\nUR:", ur_obj.type, ur_obj.cbor)

# memory control
mem_list.append(gc.mem_free())

# single-UR
ur_encoded_data = UREncoder.encode(ur_obj)
#print("\nUREncoder")
#print(ur_encoded_data)

# memory control
mem_list.append(gc.mem_free())

decoded_ur_obj = URDecoder.decode(ur_encoded_data)
#print("\nURDecoder")
#print(decoded_ur_obj.type, decoded_ur_obj.cbor)

#assert(ur_obj.type == decoded_ur_obj.type)
#assert(ur_obj.cbor == decoded_ur_obj.cbor)

# memory control
mem_list.append(gc.mem_free())

## -----------

encoder = UREncoder(ur_obj, 100, 0)

# memory control
mem_list.append(gc.mem_free())

decoder = URDecoder()
i = 0
while True:
    part = encoder.next_part()
    # every 3 parts misses 1
    if i % 3 != 0:
        decoder.receive_part(part)
        # print(decoder.fountain_decoder.received_part_indexes, decoder.fountain_decoder.processed_parts_count, decoder.fountain_decoder.expected_part_indexes, decoder.expected_part_count())
        if(decoder.is_complete()):
            break

    i+=1

#print("\nMultipart (fountain) encoder/decoder")
#print(decoder.result.type, decoder.result.cbor)

#assert ur_obj.type == decoder.result.type
#assert ur_obj.cbor == decoder.result.cbor

# -------------

time_a = time.ticks_ms()
mem_list.append(gc.mem_free())

print("data scale factor: %d - total len: %d" % (scale_factor, len(data) * scale_factor))
print("\n------\n")

print("time before:", time_b)
print("time after:", time_a)
print("diff time:", time_a - time_b)
print("\n------\n")

mem_before = mem_list[0]
for val in mem_list:
    gc_called_str = ""
    mem_diff = "diff: %d" % (mem_before - val)
    if val > mem_before and mem_before != 0:
        gc_called_str = "GC was called!"
    print("mem value:", val, mem_diff, gc_called_str)
    mem_before = val

print("diff mem (first - last):", mem_list[0] - mem_list[-1])

print("\n\n------ end foundation-ur-py test\n\n")

[tadeubas]

v0.1.0:
------ start foundation-ur-py test

data scale factor: 1 - total len: 81

---

time before: 14164
time after: 14796
diff time: 632

---

mem value: 1451776 diff: 0
mem value: 1430400 diff: 21376
mem value: 1430208 diff: 192
mem value: 1430144 diff: 64
mem value: 1429216 diff: 928
mem value: 1429056 diff: 160
mem value: 1428896 diff: 160
mem value: 1109536 diff: 319360
mem value: 1053824 diff: 55712
mem value: 1031136 diff: 22688
mem value: 868544 diff: 162592
diff mem (first - last): 583232

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 3 - total len: 729

---

time before: 6960
time after: 9145
diff time: 2185

---

mem value: 1451776 diff: 0
mem value: 1430080 diff: 21696
mem value: 1429888 diff: 192
mem value: 1429824 diff: 64
mem value: 1428576 diff: 1248
mem value: 1428256 diff: 320
mem value: 1428096 diff: 160
mem value: 978784 diff: 449312
mem value: 833312 diff: 145472
mem value: 768096 diff: 65216
mem value: 1170944 diff: -402848 GC was called!
diff mem (first - last): 280832

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 10 - total len: 8100

---

time before: 6711
time after: 11876
diff time: 5165

---

mem value: 1451776 diff: 0
mem value: 1428928 diff: 22848
mem value: 1428736 diff: 192
mem value: 1428672 diff: 64
mem value: 1426208 diff: 2464
mem value: 1425312 diff: 896
mem value: 1425152 diff: 160
mem value: 105312 diff: 1319840
mem value: 982976 diff: -877664 GC was called!
mem value: 766880 diff: 216096
mem value: 1272576 diff: -505696 GC was called!
diff mem (first - last): 179200

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 15 - total len: 18225

---

time before: 8735
time after: 16856
diff time: 8121

---

mem value: 1451776 diff: 0
mem value: 1428128 diff: 23648
mem value: 1427936 diff: 192
mem value: 1427872 diff: 64
mem value: 1424640 diff: 3232
mem value: 1423360 diff: 1280
mem value: 1423200 diff: 160
mem value: 489376 diff: 933824
mem value: 1099136 diff: -609760 GC was called!
mem value: 773664 diff: 325472
mem value: 159136 diff: 614528
diff mem (first - last): 1292640

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 20 - total len: 32400

---

time before: 12980
time after: 24700
diff time: 11720

---

mem value: 1451776 diff: 0
mem value: 1427328 diff: 24448
mem value: 1427136 diff: 192
mem value: 1427072 diff: 64
mem value: 1423008 diff: 4064
mem value: 1421312 diff: 1696
mem value: 1421152 diff: 160
mem value: 381184 diff: 1039968
mem value: 735840 diff: -354656 GC was called!
mem value: 299456 diff: 436384
mem value: 273312 diff: 26144
diff mem (first - last): 1178464

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 100 - total len: 810000

---

time before: 74838
time after: 144196
diff time: 69358

---

mem value: 1382496 diff: 0
mem value: 1365344 diff: 17152
mem value: 1365280 diff: 64
mem value: 1365216 diff: 64
mem value: 1348160 diff: 17056
mem value: 1339968 diff: 8192
mem value: 1339808 diff: 160
mem value: 1024800 diff: 315008
mem value: 627744 diff: 397056
mem value: 870240 diff: -242496 GC was called!
mem value: 77664 diff: 792576
diff mem (first - last): 1304832

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 200 - total len: 3240000

---

time before: 8950
time after: 185669
diff time: 176719

---

mem value: 1451776 diff: 0
mem value: 1398144 diff: 53632
mem value: 1397952 diff: 192
mem value: 1397888 diff: 64
mem value: 1364640 diff: 33248
mem value: 1348352 diff: 16288
mem value: 1348192 diff: 160
mem value: 1115520 diff: 232672
mem value: 131264 diff: 984256
mem value: 958272 diff: -827008 GC was called!
mem value: 1134528 diff: -176256 GC was called!
diff mem (first - last): 317248

------ end foundation-ur-py test

[tadeubas]

v2.0.0:

------ start foundation-ur-py test

data scale factor: 1 - total len: 81

---

time before: 253936
time after: 254000
diff time: 64

---

mem value: 1451776 diff: 0
mem value: 1435680 diff: 16096
mem value: 1435488 diff: 192
mem value: 1435424 diff: 64
mem value: 1434688 diff: 736
mem value: 1434528 diff: 160
mem value: 1434304 diff: 224
mem value: 1432256 diff: 2048
mem value: 1392608 diff: 39648
mem value: 1392064 diff: 544
mem value: 1335136 diff: 56928
diff mem (first - last): 116640

------ end foundation-ur-py test
------ start foundation-ur-py test

data (243) scale factor: 3 - total len: 729

---

time before: 9527
time after: 9666
diff time: 139

---

mem value: 1451776 diff: 0
mem value: 1435360 diff: 16416
mem value: 1435168 diff: 192
mem value: 1435104 diff: 64
mem value: 1434368 diff: 736
mem value: 1434048 diff: 320
mem value: 1433824 diff: 224
mem value: 1430240 diff: 3584
mem value: 1332608 diff: 97632
mem value: 1332064 diff: 544
mem value: 935456 diff: 396608
diff mem (first - last): 516320

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 10 - total len: 8100

---

time before: 535900
time after: 536252
diff time: 352

---

mem value: 1451360 diff: 0
mem value: 1433760 diff: 17600
mem value: 1433568 diff: 192
mem value: 1433504 diff: 64
mem value: 1432768 diff: 736
mem value: 1431872 diff: 896
mem value: 1431648 diff: 224
mem value: 1422976 diff: 8672
mem value: 1121952 diff: 301024
mem value: 1121408 diff: 544
mem value: 152576 diff: 968832
diff mem (first - last): 1298784

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 15 - total len: 18225

---

time before: 215864
time after: 216460
diff time: 596

---

mem value: 1451776 diff: 0
mem value: 1433408 diff: 18368
mem value: 1433216 diff: 192
mem value: 1433152 diff: 64
mem value: 1432416 diff: 736
mem value: 1431136 diff: 1280
mem value: 1430912 diff: 224
mem value: 1418624 diff: 12288
mem value: 972608 diff: 446016
mem value: 972064 diff: 544
mem value: 902464 diff: 69600
diff mem (first - last): 549312

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 20 - total len: 32400

---

time before: 7602
time after: 8549
diff time: 947

---

mem value: 1451776 diff: 0
mem value: 1432512 diff: 19264
mem value: 1432320 diff: 192
mem value: 1432256 diff: 64
mem value: 1431520 diff: 736
mem value: 1429824 diff: 1696
mem value: 1429600 diff: 224
mem value: 1413728 diff: 15872
mem value: 822784 diff: 590944
mem value: 822240 diff: 544
mem value: 1316928 diff: -494688 GC was called!
diff mem (first - last): 134848

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 100 - total len: 810000

---

time before: 7109
time after: 16748
diff time: 9639

---

mem value: 1451776 diff: 0
mem value: 1419616 diff: 32160
mem value: 1419424 diff: 192
mem value: 1419360 diff: 64
mem value: 1418624 diff: 736
mem value: 1410432 diff: 8192
mem value: 1410208 diff: 224
mem value: 1336000 diff: 74208
mem value: 1159616 diff: 176384
mem value: 1159072 diff: 544
mem value: 703552 diff: 455520
diff mem (first - last): 748224

------ end foundation-ur-py test
------ start foundation-ur-py test

data scale factor: 200 - total len: 3240000

---

time before: 64497
time after: 94464
diff time: 29967

---

mem value: 1378720 diff: 0
mem value: 1345504 diff: 33216
mem value: 1345440 diff: 64
mem value: 1345376 diff: 64
mem value: 1344640 diff: 736
mem value: 1328352 diff: 16288
mem value: 1328128 diff: 224
mem value: 1180896 diff: 147232
mem value: 481536 diff: 699360
mem value: 480992 diff: 544
mem value: 1188704 diff: -707712 GC was called!
diff mem (first - last): 190016

------ end foundation-ur-py test

[tadeubas]

Finished optimizations, values are ~10% better than tests exposed above 👍

[tadeubas]

Krux firmware changes needed:

conftests.py: UR.__eq__ was removed form the lib to reduce its size (eq was used only for tests):

# add UR eq for tests
from ur.ur import UR

def eq_for_tests(self, obj):
    if obj == None:
            return False
    return self.type == obj.type and self.cbor == obj.cbor

UR.__eq__ = eq_for_tests

datum_tool.py: All string comparisons need to be UPPERCASE (lib now defaults as it is better for QR codes):

def urobj_to_data(ur_obj):
    """returns flatened data from a UR object. belongs in qr or qr_capture???"""
    import urtypes

    if ur_obj.type == "CRYPTO-BIP39":
        data = urtypes.crypto.BIP39.from_cbor(ur_obj.cbor).words
        data = " ".join(data)
    elif ur_obj.type == "CRYPTO-ACCOUNT":
        data = (
            urtypes.crypto.Account.from_cbor(ur_obj.cbor)
            .output_descriptors[0]
            .descriptor()
        )
    elif ur_obj.type == "CRYPTO-OUTPUT":
        data = urtypes.crypto.Output.from_cbor(ur_obj.cbor).descriptor()
    elif ur_obj.type == "CRYPTO-PSBT":
        data = urtypes.crypto.PSBT.from_cbor(ur_obj.cbor).data
    elif ur_obj.type == "BYTES":
        data = urtypes.bytes.Bytes.from_cbor(ur_obj.cbor).data
    else:
        data = None
    return data

test_qr.py: test_parser() needs to check for completion before assert total_count as when completed everything is cleanup, except the result:

# add this if to the two lines that check for if num == 4:
if num == 4:
    if not parser.is_complete():
        assert parser.total_count() == len(parts) * 2

test_qr.py: test_to_qrcodes() was relying on a bug from FountainEncoder.find_nominal_fragment_length not respecting max_fragment_length:

# in cases change case (FORMAT_UR, tdata.TEST_DATA_UR, 135, 26), to:
(FORMAT_UR, tdata.TEST_DATA_UR, 135, 29),

Almost all of those changes are made on this PR: selfcustody/krux#825