From f2ea68ebb80124d447207ca9979671c30796e53c Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dustin=20Wei=28=E9=9F=A6=E4=BC=9F=29?= Date: Tue, 25 Jun 2024 11:50:18 +0800 Subject: [PATCH 1/4] update documents --- README_ZH.md | 32 ++ code.py | 31 ++ readme_en.md => docs/en/API_Reference.md | 392 ++++++++---------- .../images}/BasicDataStructure.png | Bin .../images}/ConcatenatedDataStructure.png | Bin .../images}/NestedDataStructure.png | Bin ...02\350\200\203\346\211\213\345\206\214.md" | 175 ++++++++ readme.md | 219 +--------- 8 files changed, 433 insertions(+), 416 deletions(-) create mode 100644 README_ZH.md create mode 100644 code.py rename readme_en.md => docs/en/API_Reference.md (87%) rename {images => docs/images}/BasicDataStructure.png (100%) rename {images => docs/images}/ConcatenatedDataStructure.png (100%) rename {images => docs/images}/NestedDataStructure.png (100%) create mode 100644 "docs/zh/API\345\217\202\350\200\203\346\211\213\345\206\214.md" diff --git a/README_ZH.md b/README_ZH.md new file mode 100644 index 0000000..45bb79f --- /dev/null +++ b/README_ZH.md @@ -0,0 +1,32 @@ +# QuecPython 软件看门狗 + +中文 | [English](README.md) + +## 概述 + +`TLV`是`Tag(标签)`、`Length(长度)`和`Value(数值)`的简称,是一种数据序列化的格式。其结构简单、解析速度快,支持报文嵌套和顺序拼接。在串口、蓝牙甚至网络数据传输中比较常用。 + +`QpyTLV`是用Quecpython语言编写的TLV结构编解码器,支持emv tags。 + +## 用法 + +- [API 参考手册](./docs/zh/API参考手册.md) +- [示例代码](./code/demo.py) + +## 贡献 + +我们欢迎对本项目的改进做出贡献!请按照以下步骤进行贡献: + +1. Fork 此仓库。 +2. 创建一个新分支(`git checkout -b feature/your-feature`)。 +3. 提交您的更改(`git commit -m 'Add your feature'`)。 +4. 推送到分支(`git push origin feature/your-feature`)。 +5. 打开一个 Pull Request。 + +## 许可证 + +本项目使用 Apache 许可证。详细信息请参阅 [LICENSE](LICENSE) 文件。 + +## 支持 + +如果您有任何问题或需要支持,请参阅 [QuecPython 文档](https://python.quectel.com/doc) 或在本仓库中打开一个 issue。 \ No newline at end of file diff --git a/code.py b/code.py new file mode 100644 index 0000000..c3df99b --- /dev/null +++ b/code.py @@ -0,0 +1,31 @@ +# Import module from usr +from usr.qpytlv import QpyTLV + +# Define your tags list or dict if necessary +tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] + +# Create a object of class QpyTLV +tlv = QpyTLV(tags) + +# Pack user data as a specific format dict +d = { + "aaaa": b'\xaa\xaa', + "bbbb": { + "cccc": b'\xcc\xcc', + "dddd": b'\xdd\xdd' + }, + "eeee": { + "ffff": b'\xff\xff', + "a5a5": { + "e1e1": b'\xe1\xe1' + } + } +} + +# Build a TLV structure +b = tlv.build(d) +print(b) + +# Parse a TLV structure +d = tlv.parse(b) +print(d) diff --git a/readme_en.md b/docs/en/API_Reference.md similarity index 87% rename from readme_en.md rename to docs/en/API_Reference.md index 5aa85d2..a4c6e2a 100644 --- a/readme_en.md +++ b/docs/en/API_Reference.md @@ -1,215 +1,177 @@ -# QpyTLV - Quecpython's data serialization format - -[[中文](./readme.md)] - -## Overview - -`TLV` stands for `Tag`, `Length` and `Value`, which is a data serialization format. It features a simple structure, fast parsing speed, supports nested messages, and sequential concatenation. It is commonly used in serial communication, Bluetooth, and even network data transmission. - -`QpyTLV` is a TLV structure encoder and decoder written in the Quecpython language, which also supports EMV tags. - -## Data structure - -### **Basic data structure** - -The basic data structure is as follows: - -![BasicDataStructure.png](./images/BasicDataStructure.png) - -In the diagram, the storage length of `Tag` is 2 bytes, the storage length of `Length` is 2 bytes, and the storage length of `Value` is the size specified by the `Length` value, in bytes. - -### **Nested data structure** - -The nested data structure is as follows: - -![NestedDataStructure.png](./images/NestedDataStructure.png) - -In the diagram, the nested second-level TLV structure is the value part of its higher-level structure. - -### **Concatenated data structure** - -The concatenated data structure is as follows: - -![ConcatenatedDataStructure.png](./images/ConcatenatedDataStructure.png) - -In the diagram, there are two sets of TLV data structures arranged in parallel. - -> Any complete TLV structure can serve as the value part of another TLV. - -## How to use - -### **Step1. Put script files to Quectel's module** - -Import the three script files involved in QpyTLV, namely `qpytlv.py`, `TLV.py`, and `OrderedDict.py`, into the file system of the Quectel module using the QPYcom tool. - -> - `qpytlv.py`: User API file that implements the construction and parsing of nested TLV structures. -> - `TLV.py`: Parsing of the basic TLV structure. Users can use this to implement parsing for complex TLV structures. -> - `OrderedDict.py`: Implementation of ordered dictionaries, supplementing Quecpython's lack of support for OrderedDict. Used in TLV parsing with ordered dictionaries. - -### **Step2. Import module from usr** - -```python -from usr.qpytlv import QpyTLV -``` - -### **Step3. Define your tags list or dict if necessary** - -- Define as a list - -```python -tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] -``` - -- Define as a dictionary - -```python -tags = { - 'aaaa': 'description for tag "aaaa"', - 'bbbb': 'description for tag "bbbb"', - 'cccc': 'description for tag "cccc"', - 'dddd': 'description for tag "dddd"', - 'eeee': 'description for tag "eeee"', - 'ffff': 'description for tag "ffff"', - 'a5a5': 'description for tag "a5a5"', - 'e1e1': 'description for tag "e1e1"' -} -``` - -> - QpyTLV requires users to specify business-related tags when creating objects. During parsing, these tags are used to match the tag field in the data structure. If there is a mismatch, the structure is considered invalid. -> - Tags can be defined using either of the two methods above. When defining tags in dictionary format, the key is the tag itself, and the value is a description of the tag. -> - Tags are limited to even-length hexadecimal strings with no fixed length. For example, `'06'` and `'abcd'` are valid tags, while `'6'` and `'abcx'` are not. -> - The reason tag length is not restricted here is that the length of the tag is stored at the lower level. If variable-length tags create inconvenience in business parsing, the tag size can be kept consistent with convenient parsing when defining tags. -> - Tags are constrained as strings during definition for user convenience. During the actual construction of TLV structures, they are converted to bytes, for example, `'abc'` would be converted to `b'\x0a\xbc'`. Note that this conversion is done according to the order in which the characters are written, i.e., "big-endian mode". - -### **Step4. Create a object of class QpyTLV** - -```python -tlv = QpyTLV(tags) -``` - -> If no parameters are passed when creating the object, the default EMV tags are used, see [`TLV.py`](./TLV.py) for details. - -### **Step5. Pack user data as a specific format dict** - -The basic format of user data is: -```python -{ - 'hex_tag': b'value' -} -``` - -This structure can be nested to build nested TLV structures, as shown below: -```python -{ - 'hex_tag1': { - 'hex_tag2': b'value2' - } -} -``` - -It can also be extended in parallel to build concatenated TLV structures, as shown below: -```python -{ - 'hex_tag1': b'value1', - 'hex_tag2': b'value2' -} -``` - -Based on the above rules, define user data as follows: -```python -d = { - "aaaa": b'\xaa\xaa', - "bbbb": { - "cccc": b'\xcc\xcc', - "dddd": b'\xdd\xdd' - }, - "eeee": { - "ffff": b'\xff\xff', - "a5a5": { - "e1e1": b'\xe1\xe1' - } - } -} -``` - -> User data must be in the bytes format. - -### **Step6. Build a TLV structure** - -```python -b = tlv.build(d) -print(b) -``` - -The above code constructs the TLV structure of the user data `d` into a bytes-type data `b`, which appears as follows: -```python -b'\xaa\xaa\x00\x02\xaa\xaa\xbb\xbb\x00\x0c\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc\xee\xee\x00\x10\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff' -``` - -> Upon careful observation, it can be noticed that the order of some parallel parts in the constructed TLV structure may differ slightly from the order defined in the original user data `d`. This is because the dictionary-type data `d` written by the user is not strictly stored in the order of writing, but rather arranged in ascending order of hash values within Quecpython. Printing `d` would reveal the actual output as follows: -> ```python -> {'aaaa': b'\xaa\xaa', 'bbbb': {'dddd': b'\xdd\xdd', 'cccc': b'\xcc\xcc'}, 'eeee': {'a5a5': {'e1e1': b'\xe1\xe1'}, 'ffff': b'\xff\xff'}} -> ``` -> This order is consistent with the constructed data `b`. -> -> **The recipient of the data should not equate the order of the parsed parallel TLV structures with their chronological order. For strict ordering requirements, multiple TLV structures can be constructed.** -> -> One more thing to note is that the `build` interface directly modifies the data passed as a parameter during the process of building the TLV. After the interface returns, the original parameter is no longer the same. This is done to save memory. If you don't want the parameter to be modified, you can create a new dictionary based on the original value and pass it to the `build` interface, like `tlv.build(dict(d))`. - -### **Step7. Parse a TLV structure** - -```python -d = tlv.parse(b) -print(d) -``` - -Assuming the previously constructed data `b` is the TLV structure received by the user, using the code above to parse it would result in: - -```python -[('aaaa', b'\xaa\xaa'), ('bbbb', b'\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc'), ('eeee', b'\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff')] -``` - -Users might be surprised to see that the parsed result seems quite different from the original data `d`. There are two reasons for this: - -- The output uses the OrderedDict format instead of dict, to maintain consistency with the order of the TLV structure. -- Only the first level of TLV is parsed, so only parallel structures can be parsed. QpyTLV cannot determine whether the data carried by the first level is a nested substructure or the actual user data. Upon receiving the parsed data, the business logic should call the `parse` interface multiple times for nested structure parsing. - -> The parsed result still has the tag part converted into corresponding hexadecimal strings, while the value part remains in bytes format. - -## Example - -Here's the complete example code: - -```python -# Import module from usr -from usr.qpytlv import QpyTLV - -# Define your tags list or dict if necessary -tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] - -# Create a object of class QpyTLV -tlv = QpyTLV(tags) - -# Pack user data as a specific format dict -d = { - "aaaa": b'\xaa\xaa', - "bbbb": { - "cccc": b'\xcc\xcc', - "dddd": b'\xdd\xdd' - }, - "eeee": { - "ffff": b'\xff\xff', - "a5a5": { - "e1e1": b'\xe1\xe1' - } - } -} - -# Build a TLV structure -b = tlv.build(d) -print(b) - -# Parse a TLV structure -d = tlv.parse(b) -print(d) -``` \ No newline at end of file +# QpyTLV - Quecpython's data serialization format + +[[中文](./readme.md)] + +## Overview + +`TLV` stands for `Tag`, `Length` and `Value`, which is a data serialization format. It features a simple structure, fast parsing speed, supports nested messages, and sequential concatenation. It is commonly used in serial communication, Bluetooth, and even network data transmission. + +`QpyTLV` is a TLV structure encoder and decoder written in the Quecpython language, which also supports EMV tags. + +## Data structure + +### **Basic data structure** + +The basic data structure is as follows: + +![BasicDataStructure.png](../images/BasicDataStructure.png) + +In the diagram, the storage length of `Tag` is 2 bytes, the storage length of `Length` is 2 bytes, and the storage length of `Value` is the size specified by the `Length` value, in bytes. + +### **Nested data structure** + +The nested data structure is as follows: + +![NestedDataStructure.png](../images/NestedDataStructure.png) + +In the diagram, the nested second-level TLV structure is the value part of its higher-level structure. + +### **Concatenated data structure** + +The concatenated data structure is as follows: + +![ConcatenatedDataStructure.png](../images/ConcatenatedDataStructure.png) + +In the diagram, there are two sets of TLV data structures arranged in parallel. + +> Any complete TLV structure can serve as the value part of another TLV. + +## How to use + +### **Step1. Put script files to Quectel's module** + +Import the three script files involved in QpyTLV, namely `qpytlv.py`, `TLV.py`, and `OrderedDict.py`, into the file system of the Quectel module using the QPYcom tool. + +> - `qpytlv.py`: User API file that implements the construction and parsing of nested TLV structures. +> - `TLV.py`: Parsing of the basic TLV structure. Users can use this to implement parsing for complex TLV structures. +> - `OrderedDict.py`: Implementation of ordered dictionaries, supplementing Quecpython's lack of support for OrderedDict. Used in TLV parsing with ordered dictionaries. + +### **Step2. Import module from usr** + +```python +from usr.qpytlv import QpyTLV +``` + +### **Step3. Define your tags list or dict if necessary** + +- Define as a list + +```python +tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] +``` + +- Define as a dictionary + +```python +tags = { + 'aaaa': 'description for tag "aaaa"', + 'bbbb': 'description for tag "bbbb"', + 'cccc': 'description for tag "cccc"', + 'dddd': 'description for tag "dddd"', + 'eeee': 'description for tag "eeee"', + 'ffff': 'description for tag "ffff"', + 'a5a5': 'description for tag "a5a5"', + 'e1e1': 'description for tag "e1e1"' +} +``` + +> - QpyTLV requires users to specify business-related tags when creating objects. During parsing, these tags are used to match the tag field in the data structure. If there is a mismatch, the structure is considered invalid. +> - Tags can be defined using either of the two methods above. When defining tags in dictionary format, the key is the tag itself, and the value is a description of the tag. +> - Tags are limited to even-length hexadecimal strings with no fixed length. For example, `'06'` and `'abcd'` are valid tags, while `'6'` and `'abcx'` are not. +> - The reason tag length is not restricted here is that the length of the tag is stored at the lower level. If variable-length tags create inconvenience in business parsing, the tag size can be kept consistent with convenient parsing when defining tags. +> - Tags are constrained as strings during definition for user convenience. During the actual construction of TLV structures, they are converted to bytes, for example, `'abc'` would be converted to `b'\x0a\xbc'`. Note that this conversion is done according to the order in which the characters are written, i.e., "big-endian mode". + +### **Step4. Create a object of class QpyTLV** + +```python +tlv = QpyTLV(tags) +``` + +> If no parameters are passed when creating the object, the default EMV tags are used, see [`TLV.py`](./TLV.py) for details. + +### **Step5. Pack user data as a specific format dict** + +The basic format of user data is: +```python +{ + 'hex_tag': b'value' +} +``` + +This structure can be nested to build nested TLV structures, as shown below: +```python +{ + 'hex_tag1': { + 'hex_tag2': b'value2' + } +} +``` + +It can also be extended in parallel to build concatenated TLV structures, as shown below: +```python +{ + 'hex_tag1': b'value1', + 'hex_tag2': b'value2' +} +``` + +Based on the above rules, define user data as follows: +```python +d = { + "aaaa": b'\xaa\xaa', + "bbbb": { + "cccc": b'\xcc\xcc', + "dddd": b'\xdd\xdd' + }, + "eeee": { + "ffff": b'\xff\xff', + "a5a5": { + "e1e1": b'\xe1\xe1' + } + } +} +``` + +> User data must be in the bytes format. + +### **Step6. Build a TLV structure** + +```python +b = tlv.build(d) +print(b) +``` + +The above code constructs the TLV structure of the user data `d` into a bytes-type data `b`, which appears as follows: +```python +b'\xaa\xaa\x00\x02\xaa\xaa\xbb\xbb\x00\x0c\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc\xee\xee\x00\x10\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff' +``` + +> Upon careful observation, it can be noticed that the order of some parallel parts in the constructed TLV structure may differ slightly from the order defined in the original user data `d`. This is because the dictionary-type data `d` written by the user is not strictly stored in the order of writing, but rather arranged in ascending order of hash values within Quecpython. Printing `d` would reveal the actual output as follows: +> ```python +> {'aaaa': b'\xaa\xaa', 'bbbb': {'dddd': b'\xdd\xdd', 'cccc': b'\xcc\xcc'}, 'eeee': {'a5a5': {'e1e1': b'\xe1\xe1'}, 'ffff': b'\xff\xff'}} +> ``` +> This order is consistent with the constructed data `b`. +> +> **The recipient of the data should not equate the order of the parsed parallel TLV structures with their chronological order. For strict ordering requirements, multiple TLV structures can be constructed.** +> +> One more thing to note is that the `build` interface directly modifies the data passed as a parameter during the process of building the TLV. After the interface returns, the original parameter is no longer the same. This is done to save memory. If you don't want the parameter to be modified, you can create a new dictionary based on the original value and pass it to the `build` interface, like `tlv.build(dict(d))`. + +### **Step7. Parse a TLV structure** + +```python +d = tlv.parse(b) +print(d) +``` + +Assuming the previously constructed data `b` is the TLV structure received by the user, using the code above to parse it would result in: + +```python +[('aaaa', b'\xaa\xaa'), ('bbbb', b'\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc'), ('eeee', b'\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff')] +``` + +Users might be surprised to see that the parsed result seems quite different from the original data `d`. There are two reasons for this: + +- The output uses the OrderedDict format instead of dict, to maintain consistency with the order of the TLV structure. +- Only the first level of TLV is parsed, so only parallel structures can be parsed. QpyTLV cannot determine whether the data carried by the first level is a nested substructure or the actual user data. Upon receiving the parsed data, the business logic should call the `parse` interface multiple times for nested structure parsing. + +> The parsed result still has the tag part converted into corresponding hexadecimal strings, while the value part remains in bytes format. diff --git a/images/BasicDataStructure.png b/docs/images/BasicDataStructure.png similarity index 100% rename from images/BasicDataStructure.png rename to docs/images/BasicDataStructure.png diff --git a/images/ConcatenatedDataStructure.png b/docs/images/ConcatenatedDataStructure.png similarity index 100% rename from images/ConcatenatedDataStructure.png rename to docs/images/ConcatenatedDataStructure.png diff --git a/images/NestedDataStructure.png b/docs/images/NestedDataStructure.png similarity index 100% rename from images/NestedDataStructure.png rename to docs/images/NestedDataStructure.png diff --git "a/docs/zh/API\345\217\202\350\200\203\346\211\213\345\206\214.md" "b/docs/zh/API\345\217\202\350\200\203\346\211\213\345\206\214.md" new file mode 100644 index 0000000..94c92bb --- /dev/null +++ "b/docs/zh/API\345\217\202\350\200\203\346\211\213\345\206\214.md" @@ -0,0 +1,175 @@ +# QpyTLV - Quecpython's data serialization format + +## Overview + +`TLV`是`Tag(标签)`、`Length(长度)`和`Value(数值)`的简称,是一种数据序列化的格式。其结构简单、解析速度快,支持报文嵌套和顺序拼接。在串口、蓝牙甚至网络数据传输中比较常用。 + +`QpyTLV`是用Quecpython语言编写的TLV结构编解码器,支持emv tags。 + +## Data structure + +### **Basic data structure** + +基础的数据结构如下: + +![BasicDataStructure.png](../images/BasicDataStructure.png) + +图中,`Tag`的存储长度为2字节,`Length`的存储长度为2字节,`Value`的存储长度为`Length`数值所指定的大小,单位为字节。 + +### **Nested data structure** + +嵌套的数据结构如下: + +![NestedDataStructure.png](../images/NestedDataStructure.png) + +图中嵌套的第二级TLV结构是其上一级结构的数值部分。 + +### **Concatenated data structure** + +拼接式数据结构如下: + +![ConcatenatedDataStructure.png](../images/ConcatenatedDataStructure.png) + +图中是两组前后并列关系的TLV数据结构。 + +> 任何一个完整结构的TLV都可以作为另一个TLV的数值部分。 + +## How to use + +### **Step1. Put script files to Quectel's module** + +将QpyTLV编辑码涉及的三个脚本文件`qpytlv.py`、`TLV.py`和`OrderedDict.py`通过Qpycom工具导入到移远模块的文件系统中。 + +> - `qpytlv.py`:用户API文件,实现了TLV结构的嵌套构建和解析。 +> - `TLV.py`:TLV基础结构的解析,用户可以基于此实现自己对复杂TLV结构的解析。 +> - `OrderedDict.py`:有序字典的实现,对Quecpython不支持OrderedDict的补充,TLV解析中用到有序字典。 + +### **Step2. Import module from usr** + +```python +from usr.qpytlv import QpyTLV +``` + +### **Step3. Define your tags list or dict if necessary** + +- 以list的形式定义 + +```python +tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] +``` + +- 以dict的形式定义 + +```python +tags = { + 'aaaa': 'description for tag "aaaa"', + 'bbbb': 'description for tag "bbbb"', + 'cccc': 'description for tag "cccc"', + 'dddd': 'description for tag "dddd"', + 'eeee': 'description for tag "eeee"', + 'ffff': 'description for tag "ffff"', + 'a5a5': 'description for tag "a5a5"', + 'e1e1': 'description for tag "e1e1"' +} +``` + +> - QpyTLV需要用户在创建对象时,指定业务上的tags,在解析时,用于和数据结构中的tag字段进行匹配,匹配失败后,则认为是无效数据结构。 +> - tag可使用以上两种方式进行定义,以dict的格式定义时,key值是tag本身,value值是对tag的描述。 +> - tag限制为偶数长度十六进制字符串,长度不做限制。比如`'06'`、`'abcd'`等都是合法tag,`'6'`、`'abcx'`等都是非法tag。 +> - 这里tag长度不做限制,是因为底层存储了tag的长度。如果不定长的tag给业务解析带来了不便,则业务上定义tag时,将长度保持为方便解析的大小即可。 +> - 为了方便用户书写,这里才将tag在定义时约束为字符串,实际构建TLV结构时,是将其转换为bytes类型的,比如`'abc'`会转换为`b'\x0a\xbc'`。注意这里是按照字符串的书写顺序转换的,即`大端模式`。 + +### **Step4. Create a object of class QpyTLV** + +```python +tlv = QpyTLV(tags) +``` + +> 如果创建对象时,不传入参数,则使用默认的emv_tgas,详见`TLV.py`。 + +### **Step5. Pack user data as a specific format dict** + +用户数据的基础格式为: +```python +{ + 'hex_tag': b'value' +} +``` + +以上结构可以做嵌套,用于构建嵌套的TLV结构,如下所示: +```python +{ + 'hex_tag1': { + 'hex_tag2': b'value2' + } +} +``` + +也可以做并列扩展,用于构建拼接的TLV结构,如下所示: +```python +{ + 'hex_tag1': b'value1', + 'hex_tag2': b'value2' +} +``` + +基于以上规则,定义如下用户数据: +```python +d = { + "aaaa": b'\xaa\xaa', + "bbbb": { + "cccc": b'\xcc\xcc', + "dddd": b'\xdd\xdd' + }, + "eeee": { + "ffff": b'\xff\xff', + "a5a5": { + "e1e1": b'\xe1\xe1' + } + } +} +``` + +> **用户数据全部限制为bytes格式**。 + +### **Step6. Build a TLV structure** + +```python +b = tlv.build(d) +print(b) +``` + +以上代码将用户数据`d`构建为TLV结构的bytes类型数据`b`,数值如下: +```python +b'\xaa\xaa\x00\x02\xaa\xaa\xbb\xbb\x00\x0c\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc\xee\xee\x00\x10\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff' +``` + +> 细心观察会发现,构建出的TLV结构,并列的部分,有些顺序和用户数据`d`中定义的顺序不太一样,因为用户书写的dict类型的数据`d`,在Quecpython中的存储不是严格按照用户书写顺序而来,而是按照哈希值的大小顺序排列的,使用`print(d)`打印,会发现实际输出的结果为: +> ```python +> {'aaaa': b'\xaa\xaa', 'bbbb': {'dddd': b'\xdd\xdd', 'cccc': b'\xcc\xcc'}, 'eeee': {'a5a5': {'e1e1': b'\xe1\xe1'}, 'ffff': b'\xff\xff'}} +> ``` +> 这个顺序就和构建出的数据`b`是一致的。 +> +> **数据接收方收到并列的TLV结构,不能将其对等为时间的先后顺序,对先后顺序有严格要求的可构建多个TLV结构。** +> +> **仍需注意的一点是:`build`接口在构建TLV的过程中,是直接对传入的参数进行数据修改的,接口返回后,传入参数已经不是原来的数据了,这么做是为了节约内存。如果用户不希望参数被修改,可以基于参数原始值创建一个新的dict传入到`build`接口,如`tlv.build(dict(d))`** + +### **Step7. Parse a TLV structure** + +```python +d = tlv.parse(b) +print(d) +``` + +假设上面构建出的数据`b`就是用户接收到的TLV结构,使用以上代码进行解析,结果如下: + +```python +[('aaaa', b'\xaa\xaa'), ('bbbb', b'\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc'), ('eeee', b'\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff')] +``` + +用户可能会惊奇的发现,这里解析出的结果和原始数据`d`看起来千差万别,这里有两方面的原因: + +- 使用了OrderedDict格式作为输出,而不是dict,目的是为了和TLV结构中的顺序保持一致 +- 只对第一层级的TLV进行了解析,只能解析出并列结构。因为QpyTLV无法得知第一层级携带的数据部分是嵌套的子结构还是用户数据本身。业务上在拿到解析后的数据后,根据业务需要多次调用`parse`接口进行嵌套结构的解析。 + +> 解析后的结果,tag部分仍然转换为对应的十六进制字符串,数值部分是bytes类型。 diff --git a/readme.md b/readme.md index e043c07..fa4b27a 100644 --- a/readme.md +++ b/readme.md @@ -1,215 +1,32 @@ -# QpyTLV - Quecpython's data serialization format +# QuecPython Software Watchdog -[[English](./readme_en.md)] +[中文](README_ZH.md) | English ## Overview -`TLV`是`Tag(标签)`、`Length(长度)`和`Value(数值)`的简称,是一种数据序列化的格式。其结构简单、解析速度快,支持报文嵌套和顺序拼接。在串口、蓝牙甚至网络数据传输中比较常用。 +`TLV` stands for `Tag`, `Length` and `Value`, which is a data serialization format. It features a simple structure, fast parsing speed, supports nested messages, and sequential concatenation. It is commonly used in serial communication, Bluetooth, and even network data transmission. -`QpyTLV`是用Quecpython语言编写的TLV结构编解码器,支持emv tags。 +`QpyTLV` is a TLV structure encoder and decoder written in the Quecpython language, which also supports EMV tags. -## Data structure +## Usage -### **Basic data structure** +- [API Reference Manual](./docs/en/API_Reference.md) +- [Example Code](./code/demo.py) -基础的数据结构如下: +## Contribution -![BasicDataStructure.png](./images/BasicDataStructure.png) +We welcome contributions to improve this project! Please follow these steps to contribute: -图中,`Tag`的存储长度为2字节,`Length`的存储长度为2字节,`Value`的存储长度为`Length`数值所指定的大小,单位为字节。 +1. Fork the repository. +2. Create a new branch (`git checkout -b feature/your-feature`). +3. Commit your changes (`git commit -m 'Add your feature'`). +4. Push to the branch (`git push origin feature/your-feature`). +5. Open a Pull Request. -### **Nested data structure** +## License -嵌套的数据结构如下: +This project is licensed under the Apache License. See the [LICENSE](LICENSE) file for details. -![NestedDataStructure.png](./images/NestedDataStructure.png) +## Support -图中嵌套的第二级TLV结构是其上一级结构的数值部分。 - -### **Concatenated data structure** - -拼接式数据结构如下: - -![ConcatenatedDataStructure.png](./images/ConcatenatedDataStructure.png) - -图中是两组前后并列关系的TLV数据结构。 - -> 任何一个完整结构的TLV都可以作为另一个TLV的数值部分。 - -## How to use - -### **Step1. Put script files to Quectel's module** - -将QpyTLV编辑码涉及的三个脚本文件`qpytlv.py`、`TLV.py`和`OrderedDict.py`通过Qpycom工具导入到移远模块的文件系统中。 - -> - `qpytlv.py`:用户API文件,实现了TLV结构的嵌套构建和解析。 -> - `TLV.py`:TLV基础结构的解析,用户可以基于此实现自己对复杂TLV结构的解析。 -> - `OrderedDict.py`:有序字典的实现,对Quecpython不支持OrderedDict的补充,TLV解析中用到有序字典。 - -### **Step2. Import module from usr** - -```python -from usr.qpytlv import QpyTLV -``` - -### **Step3. Define your tags list or dict if necessary** - -- 以list的形式定义 - -```python -tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] -``` - -- 以dict的形式定义 - -```python -tags = { - 'aaaa': 'description for tag "aaaa"', - 'bbbb': 'description for tag "bbbb"', - 'cccc': 'description for tag "cccc"', - 'dddd': 'description for tag "dddd"', - 'eeee': 'description for tag "eeee"', - 'ffff': 'description for tag "ffff"', - 'a5a5': 'description for tag "a5a5"', - 'e1e1': 'description for tag "e1e1"' -} -``` - -> - QpyTLV需要用户在创建对象时,指定业务上的tags,在解析时,用于和数据结构中的tag字段进行匹配,匹配失败后,则认为是无效数据结构。 -> - tag可使用以上两种方式进行定义,以dict的格式定义时,key值是tag本身,value值是对tag的描述。 -> - tag限制为偶数长度十六进制字符串,长度不做限制。比如`'06'`、`'abcd'`等都是合法tag,`'6'`、`'abcx'`等都是非法tag。 -> - 这里tag长度不做限制,是因为底层存储了tag的长度。如果不定长的tag给业务解析带来了不便,则业务上定义tag时,将长度保持为方便解析的大小即可。 -> - 为了方便用户书写,这里才将tag在定义时约束为字符串,实际构建TLV结构时,是将其转换为bytes类型的,比如`'abc'`会转换为`b'\x0a\xbc'`。注意这里是按照字符串的书写顺序转换的,即`大端模式`。 - -### **Step4. Create a object of class QpyTLV** - -```python -tlv = QpyTLV(tags) -``` - -> 如果创建对象时,不传入参数,则使用默认的emv_tgas,详见`TLV.py`。 - -### **Step5. Pack user data as a specific format dict** - -用户数据的基础格式为: -```python -{ - 'hex_tag': b'value' -} -``` - -以上结构可以做嵌套,用于构建嵌套的TLV结构,如下所示: -```python -{ - 'hex_tag1': { - 'hex_tag2': b'value2' - } -} -``` - -也可以做并列扩展,用于构建拼接的TLV结构,如下所示: -```python -{ - 'hex_tag1': b'value1', - 'hex_tag2': b'value2' -} -``` - -基于以上规则,定义如下用户数据: -```python -d = { - "aaaa": b'\xaa\xaa', - "bbbb": { - "cccc": b'\xcc\xcc', - "dddd": b'\xdd\xdd' - }, - "eeee": { - "ffff": b'\xff\xff', - "a5a5": { - "e1e1": b'\xe1\xe1' - } - } -} -``` - -> **用户数据全部限制为bytes格式**。 - -### **Step6. Build a TLV structure** - -```python -b = tlv.build(d) -print(b) -``` - -以上代码将用户数据`d`构建为TLV结构的bytes类型数据`b`,数值如下: -```python -b'\xaa\xaa\x00\x02\xaa\xaa\xbb\xbb\x00\x0c\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc\xee\xee\x00\x10\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff' -``` - -> 细心观察会发现,构建出的TLV结构,并列的部分,有些顺序和用户数据`d`中定义的顺序不太一样,因为用户书写的dict类型的数据`d`,在Quecpython中的存储不是严格按照用户书写顺序而来,而是按照哈希值的大小顺序排列的,使用`print(d)`打印,会发现实际输出的结果为: -> ```python -> {'aaaa': b'\xaa\xaa', 'bbbb': {'dddd': b'\xdd\xdd', 'cccc': b'\xcc\xcc'}, 'eeee': {'a5a5': {'e1e1': b'\xe1\xe1'}, 'ffff': b'\xff\xff'}} -> ``` -> 这个顺序就和构建出的数据`b`是一致的。 -> -> **数据接收方收到并列的TLV结构,不能将其对等为时间的先后顺序,对先后顺序有严格要求的可构建多个TLV结构。** -> -> **仍需注意的一点是:`build`接口在构建TLV的过程中,是直接对传入的参数进行数据修改的,接口返回后,传入参数已经不是原来的数据了,这么做是为了节约内存。如果用户不希望参数被修改,可以基于参数原始值创建一个新的dict传入到`build`接口,如`tlv.build(dict(d))`** - -### **Step7. Parse a TLV structure** - -```python -d = tlv.parse(b) -print(d) -``` - -假设上面构建出的数据`b`就是用户接收到的TLV结构,使用以上代码进行解析,结果如下: - -```python -[('aaaa', b'\xaa\xaa'), ('bbbb', b'\xdd\xdd\x00\x02\xdd\xdd\xcc\xcc\x00\x02\xcc\xcc'), ('eeee', b'\xa5\xa5\x00\x06\xe1\xe1\x00\x02\xe1\xe1\xff\xff\x00\x02\xff\xff')] -``` - -用户可能会惊奇的发现,这里解析出的结果和原始数据`d`看起来千差万别,这里有两方面的原因: - -- 使用了OrderedDict格式作为输出,而不是dict,目的是为了和TLV结构中的顺序保持一致 -- 只对第一层级的TLV进行了解析,只能解析出并列结构。因为QpyTLV无法得知第一层级携带的数据部分是嵌套的子结构还是用户数据本身。业务上在拿到解析后的数据后,根据业务需要多次调用`parse`接口进行嵌套结构的解析。 - -> 解析后的结果,tag部分仍然转换为对应的十六进制字符串,数值部分是bytes类型。 - -## Example - -完整的示例代码如下: - -```python -# Import module from usr -from usr.qpytlv import QpyTLV - -# Define your tags list or dict if necessary -tags = ['aaaa', 'bbbb', 'cccc', 'dddd', 'eeee', 'ffff', 'a5a5', 'e1e1'] - -# Create a object of class QpyTLV -tlv = QpyTLV(tags) - -# Pack user data as a specific format dict -d = { - "aaaa": b'\xaa\xaa', - "bbbb": { - "cccc": b'\xcc\xcc', - "dddd": b'\xdd\xdd' - }, - "eeee": { - "ffff": b'\xff\xff', - "a5a5": { - "e1e1": b'\xe1\xe1' - } - } -} - -# Build a TLV structure -b = tlv.build(d) -print(b) - -# Parse a TLV structure -d = tlv.parse(b) -print(d) -``` \ No newline at end of file +If you have any questions or need support, please refer to the [QuecPython documentation](https://python.quectel.com/doc/en) or open an issue in this repository. \ No newline at end of file From 1d8d94d5d39abdb3b479075f88c06f295c772e3a Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dustin=20Wei=28=E9=9F=A6=E4=BC=9F=29?= Date: Tue, 25 Jun 2024 11:52:04 +0800 Subject: [PATCH 2/4] create code dir and demo.py --- code.py => code/demo.py | 0 docs/en/API_Reference.md | 2 -- 2 files changed, 2 deletions(-) rename code.py => code/demo.py (100%) diff --git a/code.py b/code/demo.py similarity index 100% rename from code.py rename to code/demo.py diff --git a/docs/en/API_Reference.md b/docs/en/API_Reference.md index a4c6e2a..c23d7ee 100644 --- a/docs/en/API_Reference.md +++ b/docs/en/API_Reference.md @@ -1,7 +1,5 @@ # QpyTLV - Quecpython's data serialization format -[[中文](./readme.md)] - ## Overview `TLV` stands for `Tag`, `Length` and `Value`, which is a data serialization format. It features a simple structure, fast parsing speed, supports nested messages, and sequential concatenation. It is commonly used in serial communication, Bluetooth, and even network data transmission. From 718c5a0bcbe8728706b0de66f634ef24833eb0b1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dustin=20Wei=28=E9=9F=A6=E4=BC=9F=29?= Date: Tue, 25 Jun 2024 11:54:26 +0800 Subject: [PATCH 3/4] fix wrong title --- README_ZH.md | 2 +- readme.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README_ZH.md b/README_ZH.md index 45bb79f..702b4aa 100644 --- a/README_ZH.md +++ b/README_ZH.md @@ -1,4 +1,4 @@ -# QuecPython 软件看门狗 +# Quecpython 数据序列化格式 -- QpyTLV 中文 | [English](README.md) diff --git a/readme.md b/readme.md index fa4b27a..6aeb351 100644 --- a/readme.md +++ b/readme.md @@ -1,4 +1,4 @@ -# QuecPython Software Watchdog +# Quecpython's data serialization format -- QpyTLV [中文](README_ZH.md) | English From ddccd2a41239f179641108e9727ee5e6d68678e8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Dustin=20Wei=28=E9=9F=A6=E4=BC=9F=29?= Date: Tue, 25 Jun 2024 13:04:42 +0800 Subject: [PATCH 4/4] update README.md --- readme.md => README.md | 62 +++++++++++++++++++++--------------------- 1 file changed, 31 insertions(+), 31 deletions(-) rename readme.md => README.md (97%) diff --git a/readme.md b/README.md similarity index 97% rename from readme.md rename to README.md index 6aeb351..66c441b 100644 --- a/readme.md +++ b/README.md @@ -1,32 +1,32 @@ -# Quecpython's data serialization format -- QpyTLV - -[中文](README_ZH.md) | English - -## Overview - -`TLV` stands for `Tag`, `Length` and `Value`, which is a data serialization format. It features a simple structure, fast parsing speed, supports nested messages, and sequential concatenation. It is commonly used in serial communication, Bluetooth, and even network data transmission. - -`QpyTLV` is a TLV structure encoder and decoder written in the Quecpython language, which also supports EMV tags. - -## Usage - -- [API Reference Manual](./docs/en/API_Reference.md) -- [Example Code](./code/demo.py) - -## Contribution - -We welcome contributions to improve this project! Please follow these steps to contribute: - -1. Fork the repository. -2. Create a new branch (`git checkout -b feature/your-feature`). -3. Commit your changes (`git commit -m 'Add your feature'`). -4. Push to the branch (`git push origin feature/your-feature`). -5. Open a Pull Request. - -## License - -This project is licensed under the Apache License. See the [LICENSE](LICENSE) file for details. - -## Support - +# Quecpython's data serialization format -- QpyTLV + +[中文](README_ZH.md) | English + +## Overview + +`TLV` stands for `Tag`, `Length` and `Value`, which is a data serialization format. It features a simple structure, fast parsing speed, supports nested messages, and sequential concatenation. It is commonly used in serial communication, Bluetooth, and even network data transmission. + +`QpyTLV` is a TLV structure encoder and decoder written in the Quecpython language, which also supports EMV tags. + +## Usage + +- [API Reference Manual](./docs/en/API_Reference.md) +- [Example Code](./code/demo.py) + +## Contribution + +We welcome contributions to improve this project! Please follow these steps to contribute: + +1. Fork the repository. +2. Create a new branch (`git checkout -b feature/your-feature`). +3. Commit your changes (`git commit -m 'Add your feature'`). +4. Push to the branch (`git push origin feature/your-feature`). +5. Open a Pull Request. + +## License + +This project is licensed under the Apache License. See the [LICENSE](LICENSE) file for details. + +## Support + If you have any questions or need support, please refer to the [QuecPython documentation](https://python.quectel.com/doc/en) or open an issue in this repository. \ No newline at end of file