Big Endianess

[guyou]

I declared the following packet:

big_endian_packets

// https://ccsds.org/Pubs/133x0b2e2.pdf
struct PacketPrimaryHeader {
  packet_version_number: 3,
  packet_type: 1,
  secondary_header: 1,
  apid: 11,
}

packet CcsdsSpacePacket {
  header: PacketPrimaryHeader,
}

Then I use it in the folowing Rust code:

    let c = ccsds::CcsdsSpacePacket {
        header: PacketPrimaryHeader {
            packet_version_number: 0,
            packet_type: 1,
            secondary_header: 0,
            apid: 1,
        },
    };
    let b = c.encode_to_vec().unwrap();
    let file = File::create("ccsds.bin").unwrap();
    let mut writer = BufWriter::new(file);
    writer.write_all(&b);
    writer.flush();

I'm surprised by the result

$ xxd -b ~/Devel/pdl/pdl-tests/ccsds.bin 
00000000: 00000000 00101000                                      .(

I expected

00000000: 00010000 00000001                                      ..

What's wrong in my declaration?

[guyou]

Note: I double checked with Kaitai and the following struct

meta:
  id: ccsds
  title: CCSDS Packet
  file-extension: bin
  endian: be
  bit-endian: be
seq:
  - id: packet_version_number
    type: b3
  - id: packet_type
    type: b1
  - id: secondary_header
    type: b1
  - id: apid
    type: b11

The producd ccsds.bin is not read as expected. If I manually create a file with 00010000 00000001 the Kaitai struct read fields correctly (packet_type=1, apid=1).

[hchataing]

The toplevel endianness selection is applied at octet level, not at the field declaration. i.e. it indicates if packed bitfields are byte swapped when encoded or not. The fields are always declared in little endian order.

Your declaration produces the following memory layout:

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  |            apid[10:3]                 |
//     +---------------------------------------+
//  1  | apid[2:0]    | sh | pt |  pvn         |
//     +---------------------------------------+
//
struct PacketPrimaryHeader {
  packet_version_number: 3,
  packet_type: 1,
  secondary_header: 1,
  apid: 11,
}

What I am assuming you want is:

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  | pvn          | pt | sh | apid[2:0]    |
//     +---------------------------------------+
//  1  |            apid[10:3]                 |
//     +---------------------------------------+
//
struct PacketPrimaryHeader {
  apid: 11,
  secondary_header: 1,
  packet_type: 1,
  packet_version_number: 3,
}

[guyou]

Thanks for the answer. I totally missed this information.
Does it make sense to add a new flag in order to control the order of fields?

[hchataing]

To be very honest everyone finds the endianness model counter intuitive.
It could make sense to also swap the field order when big endian is used 👍

We have also played with the idea of adding a compound field structure to make the field groups more explicit.
Do you think this is more readable ?

struct Test {
   field1 : 16, // round number of octets, compound declaration not needed.
   // compound field declaration with explicit endianness.
   be {
      field2 : 1
      field3 : 8
      field4 : 7
    }
}

[gbonnefille]

I'm not really familiar with endianess, I just know its existence and how to deal with it. But it's not my daily job. So, for a user like me, I just want to ignore it and simply declare it. So, I don't know if there is any logical way to describe the fields order. The standard I know use network endianess (big). In such situation, as I read my packet declaration from left to right, I just want to describe them top to bottom. Is it more natural to stack the field declaration in reverse order for little endian? This seems counter intuitive for me, but once again this is not my daily job.

Since you alert me on this "feature" I checked the code. I imagined a new flag field_order_declaration = { "little_endian_fields_order" | "big_endian_fields_order" }. Then, I imagined to adapt the code that list the fields in order to process them in a reverse order, if needed. But this seems to have a significant impact on the code.

[guyou]

I did some experiments with big endianess and sub-byte fiields.

big_endian_packets

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  |         b         |        a          |
//     +---------------------------------------+
//  1  |         d         |        c          |
//     +---------------------------------------+
//  2  |         f         |        e          |
//     +---------------------------------------+
//
packet Sequence {
  b: 4,
  a: 4,
  d: 4,
  c: 4,
  f: 4,
  e: 4,
}

//     MSB                                   LSB
//     7    6    5    4    3    2    1    0
//     +---------------------------------------+
//  0  |        a          |        b          |
//     +---------------------------------------+
//  1  |         b         |        c          |
//     +---------------------------------------+
//  2  |                   c                   |
//     +---------------------------------------+
//
packet Header {
    c: 12,
    b: 8,
    a: 4,
}

The user has to know many details of the implementation in order to know how to organize the struct declaration.
In the Header example, we have to invert ALL fields. But in Sequence example, we just have to invert the fields by two.