Overview

Introduction

OSER is an easy to use, flexible object oriented serializer and deserializer that can be used for translation between binary data formats and its internal structure represented by python classes.

OSER is like a toolbox. It offers many building blocks to build your data format quickly and easily.

Unlike other serializers OSER makes it possible to inspect your data instances in an hierarchical way with or without the binary data aligned to the members. In addition OSER ist capable of building conditional serializers and deserializers using oser.IfElse, oser.Switch, oser.Array and oser.String by accessing the context.

For example OSER can be used to build and parse network protocols, image data and various binary files or you can create the content for an EEPROM for an embedded system, etc..

General concepts

An OSER instance consists of a tree of other OSER instances that implement oser.OserNode. Every OSER instance sublasses the oser.OserNode. The composite pattern is used here.

Every member that does not start with _ is included in serialization and deserialization.

>>> from __future__ import \
...     absolute_import, division, print_function, unicode_literals

>>> from oser import ByteStruct
>>> from oser import UBInt8, UBInt16
>>> from oser import to_hex

>>> class SubData(ByteStruct):
...     def __init__(self):
...         super(SubData, self).__init__()
...         self.sub1 = UBInt8(1)
...         self.sub2 = UBInt16(1000)
...

>>> class Data(ByteStruct):
...     def __init__(self):
...         super(Data, self).__init__()
...         self.a = UBInt8(1)
...         self.b = UBInt16(1000)
...         self.sub = SubData()
...
>>> instance = Data()

Every OSER instance can be viewed when it is converted into a string, for example:

>>> print(instance)
Data():
    a: 1 (UBInt8)
    b: 1000 (UBInt16)
    sub: SubData():
        sub1: 1 (UBInt8)
        sub2: 1000 (UBInt16)

This shows the internal structure only. To view the internal structure and the binary structure, you can use oser.OserNode.introspect():

>>> print(instance.introspect())
   -    -  Data():
   0 \x01      a: 1 (UBInt8)
   1 \x03      b: 1000 (UBInt16)
   2 \xe8
   -    -      sub: SubData():
   3 \x01          sub1: 1 (UBInt8)
   4 \x03          sub2: 1000 (UBInt16)
   5 \xe8

To build binary data from an OSER instance, simply use oser.OserNode.encode():

>>> binary = instance.encode()
>>> print(to_hex(binary))
   0|  1|  2|  3|  4|  5
\x01\x03\xE8\x01\x03\xE8

To decode binary data into an OSER instance, simply use oser.OserNode.decode():

>>> data = b"\x01\x02\x03\x04\x05\x06"
>>> bytesDecoded = instance.decode(data)
>>> print(bytesDecoded)
6
>>> print(instance.introspect())
   -    -  Data():
   0 \x01      a: 1 (UBInt8)
   1 \x02      b: 515 (UBInt16)
   2 \x03
   -    -      sub: SubData():
   3 \x04          sub1: 4 (UBInt8)
   4 \x05          sub2: 1286 (UBInt16)
   5 \x06

To access the root element call oser.OserNode.root():

>>> root = instance.sub.sub2.root()
>>> print(root)
Data():
    a: 1 (UBInt8)
    b: 515 (UBInt16)
    sub: SubData():
        sub1: 4 (UBInt8)
        sub2: 1286 (UBInt16)

To access the upper element (parent element) call oser.OserNode.up():

>>> up = instance.sub.sub2.up()
>>> print(up)
SubData():
    sub1: 4 (UBInt8)
    sub2: 1286 (UBInt16)

>>>
>>> upup = up.up()
>>> print(upup)
Data():
    a: 1 (UBInt8)
    b: 515 (UBInt16)
    sub: SubData():
        sub1: 4 (UBInt8)
        sub2: 1286 (UBInt16)

Navigation in the element tree is useful to build conditional blocks.

OserNode

digraph inheritance8bf54634e8 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "oser.core.OserNode" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; }
class oser.OserNode
__str__(indent=0, name=None, stop_at=None)

Return the representation of the object as a string.

decode(data, full_data=b'', context_data=b'')

Decode a binary string into a byte type and return the number of bytes that were decoded.

Parameters
  • data (str) – the data buffer that is decoded.

  • full_data (str) – the binary data string until the part to be decoded. The user normally does not need to supply this.

  • context_data (str) – the binary data of the current context. The user normally does not need to supply this.

Returns

the number of bytes that were decoded.

Return type

int

encode(full_data=b'', context_data=b'')

Return the encoded binary string.

Parameters
  • full_data (bytes) – the binary data string until the part to be encoded. The user normally does not need to supply this.

  • context_data (bytes) – the binary data of the current context. The user normally does not need to supply this.

Returns

the encoded binary string.

Return type

bytes

introspect(stop_at=None)

Return the introspection representation of the object as a string.

Parameters

stop_at=None (object) – stop introspection at stop_at.

root()

Return the root element.

size()

Return the size in bytes or bits depending on the type.

up()

Return the parent element.

Accessing data

Accessing members

To access members in OSER instances simply use the dot. Your IDE is able to help you expanding the valid values. In eclipse you can press CTRL+SPACE when the curser is right behind the instance variable to get a list of valid members of an instance.

Example:

>>> from __future__ import \
...     absolute_import, division, print_function, unicode_literals

>>> from oser import ByteStruct
>>> from oser import Enum
>>> from oser import UBInt16

>>> class SubData(ByteStruct):
...     def __init__(self, *args, **kwargs):
...         ByteStruct.__init__(self, *args, **kwargs)
...
...         self.subdata = UBInt16(23)
...
>>> class Data(ByteStruct):
...     def __init__(self):
...         super(Data, self).__init__()
...         self.enum = Enum(prototype=UBInt16,
...                          values={
...                              "value1"    :    1,
...                              "value22"   :   22,
...                              "value333"  :  333,
...                              "value4444" : 4444,
...                          }, value="value22")
...
...         self.s = SubData()
...
>>> data = Data()
>>> print(data.s.subdata.get())
23
>>> data.s.subdata.set(3)
>>> print(data.s.subdata.get())
3
>>> print(data.enum.get())
value22

Reading values

Every oser.ByteType is an instance of a class. Simply use oser.ByteType.get() to get the current value.

Since arithmetic emulation is implemented for all primitive types, there is no need to call oser.ByteType.get() if you want to compare two values, etc..

Example:

>>> from __future__ import \
...     absolute_import, division, print_function, unicode_literals

>>> from oser import ByteStruct
>>> from oser import Enum
>>> from oser import UBInt16

>>> class SubData(ByteStruct):
...     def __init__(self, *args, **kwargs):
...         ByteStruct.__init__(self, *args, **kwargs)
...
...         self.subdata = UBInt16(23)
...
>>> class Data(ByteStruct):
...     def __init__(self):
...         super(Data, self).__init__()
...         self.s = SubData()
...
>>> data = Data()
>>> print(data.s.subdata.get())
23
>>> data.s.subdata.set(3)
>>> print(data.s.subdata.get())
3

Setting values

Every oser.ByteType is an instance of a class. Simply use oser.ByteType.set() to set the current value.

Example:

>>> from __future__ import \
...     absolute_import, division, print_function, unicode_literals

>>> from oser import ByteStruct
>>> from oser import Enum
>>> from oser import UBInt16
>>> from oser import to_hex

>>> class Data(ByteStruct):
...         def __init__(self):
...             super(Data, self).__init__()
...             self.enum = Enum(prototype=UBInt16,
...                              values={
...                                  "value1"    :    1,
...                                  "value22"   :   22,
...                                  "value333"  :  333,
...                                  "value4444" : 4444,
...                              }, value="value22")
...
>>> instance = Data()
>>> instance.enum.set("value1")
>>> print(instance)
Data():
    enum: 'value1' (UBInt16)

>>> print(instance.introspect())
   -    -  Data():
   0 \x00      enum: 1 (UBInt16)
   1 \x01

>>> binary = instance.encode()
>>> print(to_hex(binary))
   0|  1
\x00\x01
>>> bytesDecoded = instance.decode(binary)
>>> print(bytesDecoded)
2
>>> print(instance)
Data():
    enum: 'value1' (UBInt16)

The context

Every building block is aware of its context. Conditional building blocks use the context to decide how to proceed.

Variable length string

The simplest example is a oser.String with a variable length.

>>> from __future__ import \
...     absolute_import, division, print_function, unicode_literals

>>> from oser import ByteStruct
>>> from oser import String, Switch, Null, IfElse
>>> from oser import UBInt8, UBInt16, UBInt32, Array

>>> class VariableLengthString(ByteStruct):
...     def __init__(self):
...         super(VariableLengthString, self).__init__()
...         self.length = UBInt16(1)
...         self.data = String(
...             length=lambda self: self.length.get(),
...             value="abcdefghijklmnopqrstuvwxyz")
...
>>> instance = VariableLengthString()
>>> print(instance.introspect())
   -    -  Data():
   0 \x00      length: 1 (UBInt16)
   1 \x01
   -    -      data: String():
   2 \x61          u'a'

>>> instance.length.set(16)
>>> print(instance.introspect())
   -    -  Data():
   0 \x00      length: 16 (UBInt16)
   1 \x10
   -    -      data: String():
   2 \x61          u'a'
   3 \x62          u'b'
   4 \x63          u'c'
   5 \x64          u'd'
   6 \x65          u'e'
   7 \x66          u'f'
   8 \x67          u'g'
   9 \x68          u'h'
  10 \x69          u'i'
  11 \x6a          u'j'
  12 \x6b          u'k'
  13 \x6c          u'l'
  14 \x6d          u'm'
  15 \x6e          u'n'
  16 \x6f          u'o'
  17 \x70          u'p'

The length of the oser.String is a callable here that returns the length’s value.

For simple true-false-decisions oser.IfElse can be used.

>>> class IfElseData(ByteStruct):
...     def __init__(self):
...         super(IfElseData, self).__init__()
...         self.true_false = UBInt8(1)
...         self.data = IfElse(condition=lambda self: bool(self.true_false.get()),
...                            if_true=UBInt8(1),
...                            if_false=UBInt32(0xffffffff)
...                            )
...
>>> instance = IfElseData()
>>> print(instance.introspect())
   -    -  IfElseData():
   0 \x01      true_false: 1 (UBInt8)
   1 \x01      data: 1 (UBInt8)

>>> instance.true_false.set(0)
>>> print(instance.introspect())
   -    -  IfElseData():
   0 \x00      true_false: 0 (UBInt8)
   1 \xff      data: 4294967295 (UBInt32)
   2 \xff
   3 \xff
   4 \xff

The condition of oser.IfElse is a callable here that returns the true_false’s value as bool.

A more complex example is a oser.Switch that decides the type of the payload.

>>> class SwitchData(ByteStruct):
...     def __init__(self):
...         super(SwitchData, self).__init__()
...         self.type = UBInt8(1)
...         self.data = Switch(condition=lambda self: self.type.get(),
...                            values={
...                                1: UBInt8(1),
...                                2: UBInt16(2),
...                                4: UBInt32(3)
...                            },
...                            default=Null())
...
>>> instance = SwitchData()
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x01      type: 1 (UBInt8)
   1 \x01      data: 1 (UBInt8)

>>> instance.type.set(2)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x02      type: 2 (UBInt8)
   1 \x00      data: 2 (UBInt16)
   2 \x02

>>> instance.type.set(4)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x04      type: 4 (UBInt8)
   1 \x00      data: 3 (UBInt32)
   2 \x00
   3 \x00
   4 \x03

>>> instance.type.set(3)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x03      type: 3 (UBInt8)
   -    -      data: Null

The condition of oser.Switch is a callable here that returns the type’s value. If type is not in [1,2,4] oser.Null is used.

To access parent elements use oser.OserNode.up().

>>> class SubSwitch(ByteStruct):
...     def __init__(self):
...         super(SubSwitch, self).__init__()
...         self.data = Switch(condition=lambda self: self.up().type.get(),
...                            values={
...                                1: UBInt8(1),
...                                2: UBInt16(2),
...                                4: UBInt32(3)
...                            },
...                            default=Null())
...
>>>
>>> class SwitchData2(ByteStruct):
...     def __init__(self):
...         super(SwitchData2, self).__init__()
...         self.type = UBInt8(1)
...         self.data = SubSwitch()
...
>>> instance = SwitchData()
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x01      type: 1 (UBInt8)
   1 \x01      data: 1 (UBInt8)

>>> instance.type.set(2)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x02      type: 2 (UBInt8)
   1 \x00      data: 2 (UBInt16)
   2 \x02

>>> instance.type.set(4)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x04      type: 4 (UBInt8)
   1 \x00      data: 3 (UBInt32)
   2 \x00
   3 \x00
   4 \x03

>>> instance.type.set(3)
>>> print(instance.introspect())
   -    -  SwitchData():
   0 \x03      type: 3 (UBInt8)
   -    -      data: Null

In this example the type is accessed by self.up().type.get().

It is also possible to create variable length repeated fields using oser.Array.

>>> class VariableLengthArray(ByteStruct):
...     def __init__(self):
...         super(VariableLengthArray, self).__init__()
...         self.length = UBInt16(1)
...         self.data = Array(
...             length=lambda self: self.length.get(),
...             prototype=UBInt8,
...             values=[UBInt8(ii) for ii in range(256)])
...
>>> instance = VariableLengthArray()
>>> print(instance.introspect())
   -    -  VariableLengthArray():
   0 \x00      length: 1 (UBInt16)
   1 \x01
   -    -      data: Array():
   -    -      [
   2 \x00          @0: 0 (UBInt8)
   -    -      ]

>>> instance.length.set(5)
>>> print(instance.introspect())
   -    -  VariableLengthArray():
   0 \x00      length: 5 (UBInt16)
   1 \x05
   -    -      data: Array():
   -    -      [
   2 \x00          @0: 0 (UBInt8)
   3 \x01          @1: 1 (UBInt8)
   4 \x02          @2: 2 (UBInt8)
   5 \x03          @3: 3 (UBInt8)
   6 \x04          @4: 4 (UBInt8)
   -    -      ]

In this example length is a callable like in the variable length string example.

Copying

OSER instances can be deeply copied using copy.deepcopy(). Shallow copies are not allowed since the context of each element must be distinct. If copy.copy() is applied on an OSER instance and exception is raised. While being copied encode() is called with all its side effects.

>>> from __future__ import absolute_import, division, print_function, unicode_literals
>>>
>>> import oser
>>> import copy
>>>
>>>
>>> class Struct(oser.ByteStruct):
...     def __init__(self):
...         super(Struct, self).__init__()
...         self.a = oser.ULInt8(1)
...
>>> s = Struct()
>>> id(s)
6596088
>>> print(s)
Struct():
    a: 1 (ULInt8)

>>>
>>> deep_copy = copy.deepcopy(s)
>>> id(deep_copy)
38738520
>>>
>>> s.a.set(100)  # this does not influence the copy
>>> print(deep_copy)
Struct():
    a: 1 (ULInt8)