Overview¶
Introduction¶
OSER is an easy to use, flexible object oriented serializer and deserializer that can be used for translation between binary data formats and its internal structure represented by python classes.
OSER is like a toolbox. It offers many building blocks to build your data format quickly and easily.
Unlike other serializers OSER makes it possible to inspect your data instances in an hierarchical
way with or without the binary data aligned to the members.
In addition OSER ist capable of building conditional serializers and deserializers using
oser.IfElse
, oser.Switch
, oser.Array
and oser.String
by accessing the context.
For example OSER can be used to build and parse network protocols, image data and various binary files or you can create the content for an EEPROM for an embedded system, etc..
General concepts¶
An OSER instance consists of a tree of other OSER instances that implement oser.OserNode
.
Every OSER instance sublasses the oser.OserNode
. The composite pattern is used here.
Every member that does not start with _
is
included in serialization and deserialization.
>>> from __future__ import \
... absolute_import, division, print_function, unicode_literals
>>> from oser import ByteStruct
>>> from oser import UBInt8, UBInt16
>>> from oser import to_hex
>>> class SubData(ByteStruct):
... def __init__(self):
... super(SubData, self).__init__()
... self.sub1 = UBInt8(1)
... self.sub2 = UBInt16(1000)
...
>>> class Data(ByteStruct):
... def __init__(self):
... super(Data, self).__init__()
... self.a = UBInt8(1)
... self.b = UBInt16(1000)
... self.sub = SubData()
...
>>> instance = Data()
Every OSER instance can be viewed when it is converted into a string, for example:
>>> print(instance)
Data():
a: 1 (UBInt8)
b: 1000 (UBInt16)
sub: SubData():
sub1: 1 (UBInt8)
sub2: 1000 (UBInt16)
This shows the internal structure only.
To view the internal structure and the binary structure, you can use oser.OserNode.introspect()
:
>>> print(instance.introspect())
- - Data():
0 \x01 a: 1 (UBInt8)
1 \x03 b: 1000 (UBInt16)
2 \xe8
- - sub: SubData():
3 \x01 sub1: 1 (UBInt8)
4 \x03 sub2: 1000 (UBInt16)
5 \xe8
To build binary data from an OSER instance, simply use oser.OserNode.encode()
:
>>> binary = instance.encode()
>>> print(to_hex(binary))
0| 1| 2| 3| 4| 5
\x01\x03\xE8\x01\x03\xE8
To decode binary data into an OSER instance, simply use oser.OserNode.decode()
:
>>> data = b"\x01\x02\x03\x04\x05\x06"
>>> bytesDecoded = instance.decode(data)
>>> print(bytesDecoded)
6
>>> print(instance.introspect())
- - Data():
0 \x01 a: 1 (UBInt8)
1 \x02 b: 515 (UBInt16)
2 \x03
- - sub: SubData():
3 \x04 sub1: 4 (UBInt8)
4 \x05 sub2: 1286 (UBInt16)
5 \x06
To access the root element call oser.OserNode.root()
:
>>> root = instance.sub.sub2.root()
>>> print(root)
Data():
a: 1 (UBInt8)
b: 515 (UBInt16)
sub: SubData():
sub1: 4 (UBInt8)
sub2: 1286 (UBInt16)
To access the upper element (parent element) call oser.OserNode.up()
:
>>> up = instance.sub.sub2.up()
>>> print(up)
SubData():
sub1: 4 (UBInt8)
sub2: 1286 (UBInt16)
>>>
>>> upup = up.up()
>>> print(upup)
Data():
a: 1 (UBInt8)
b: 515 (UBInt16)
sub: SubData():
sub1: 4 (UBInt8)
sub2: 1286 (UBInt16)
Navigation in the element tree is useful to build conditional blocks.
OserNode¶
digraph inheritance8bf54634e8 { bgcolor=transparent; rankdir=LR; size="8.0, 12.0"; "oser.core.OserNode" [fillcolor=white,fontname="Vera Sans, DejaVu Sans, Liberation Sans, Arial, Helvetica, sans",fontsize=10,height=0.25,shape=box,style="setlinewidth(0.5),filled"]; }-
class
oser.
OserNode
¶ -
__str__
(indent=0, name=None, stop_at=None)¶ Return the representation of the object as a string.
-
decode
(data, full_data=b'', context_data=b'')¶ Decode a binary string into a byte type and return the number of bytes that were decoded.
- Parameters
- Returns
the number of bytes that were decoded.
- Return type
-
encode
(full_data=b'', context_data=b'')¶ Return the encoded binary string.
-
introspect
(stop_at=None)¶ Return the introspection representation of the object as a string.
- Parameters
stop_at=None (object) – stop introspection at
stop_at
.
-
root
()¶ Return the root element.
-
size
()¶ Return the size in bytes or bits depending on the type.
-
up
()¶ Return the parent element.
-
Accessing data¶
Accessing members¶
To access members in OSER instances simply use the dot
. Your IDE is able to help you expanding the valid values.
In eclipse you can press CTRL+SPACE
when the curser is right behind the
instance variable to get a list of valid members of an instance.
Example:
>>> from __future__ import \
... absolute_import, division, print_function, unicode_literals
>>> from oser import ByteStruct
>>> from oser import Enum
>>> from oser import UBInt16
>>> class SubData(ByteStruct):
... def __init__(self, *args, **kwargs):
... ByteStruct.__init__(self, *args, **kwargs)
...
... self.subdata = UBInt16(23)
...
>>> class Data(ByteStruct):
... def __init__(self):
... super(Data, self).__init__()
... self.enum = Enum(prototype=UBInt16,
... values={
... "value1" : 1,
... "value22" : 22,
... "value333" : 333,
... "value4444" : 4444,
... }, value="value22")
...
... self.s = SubData()
...
>>> data = Data()
>>> print(data.s.subdata.get())
23
>>> data.s.subdata.set(3)
>>> print(data.s.subdata.get())
3
>>> print(data.enum.get())
value22
Reading values¶
Every oser.ByteType
is an instance of a class.
Simply use oser.ByteType.get()
to get the current value.
Since arithmetic emulation is implemented for all primitive types, there is no need to call
oser.ByteType.get()
if you want to compare two values, etc..
Example:
>>> from __future__ import \
... absolute_import, division, print_function, unicode_literals
>>> from oser import ByteStruct
>>> from oser import Enum
>>> from oser import UBInt16
>>> class SubData(ByteStruct):
... def __init__(self, *args, **kwargs):
... ByteStruct.__init__(self, *args, **kwargs)
...
... self.subdata = UBInt16(23)
...
>>> class Data(ByteStruct):
... def __init__(self):
... super(Data, self).__init__()
... self.s = SubData()
...
>>> data = Data()
>>> print(data.s.subdata.get())
23
>>> data.s.subdata.set(3)
>>> print(data.s.subdata.get())
3
Setting values¶
Every oser.ByteType
is an instance of a class.
Simply use oser.ByteType.set()
to set the current value.
Example:
>>> from __future__ import \
... absolute_import, division, print_function, unicode_literals
>>> from oser import ByteStruct
>>> from oser import Enum
>>> from oser import UBInt16
>>> from oser import to_hex
>>> class Data(ByteStruct):
... def __init__(self):
... super(Data, self).__init__()
... self.enum = Enum(prototype=UBInt16,
... values={
... "value1" : 1,
... "value22" : 22,
... "value333" : 333,
... "value4444" : 4444,
... }, value="value22")
...
>>> instance = Data()
>>> instance.enum.set("value1")
>>> print(instance)
Data():
enum: 'value1' (UBInt16)
>>> print(instance.introspect())
- - Data():
0 \x00 enum: 1 (UBInt16)
1 \x01
>>> binary = instance.encode()
>>> print(to_hex(binary))
0| 1
\x00\x01
>>> bytesDecoded = instance.decode(binary)
>>> print(bytesDecoded)
2
>>> print(instance)
Data():
enum: 'value1' (UBInt16)
The context¶
Every building block is aware of its context. Conditional building blocks use the context to decide how to proceed.
Variable length string¶
The simplest example is a oser.String
with a variable length.
>>> from __future__ import \
... absolute_import, division, print_function, unicode_literals
>>> from oser import ByteStruct
>>> from oser import String, Switch, Null, IfElse
>>> from oser import UBInt8, UBInt16, UBInt32, Array
>>> class VariableLengthString(ByteStruct):
... def __init__(self):
... super(VariableLengthString, self).__init__()
... self.length = UBInt16(1)
... self.data = String(
... length=lambda self: self.length.get(),
... value="abcdefghijklmnopqrstuvwxyz")
...
>>> instance = VariableLengthString()
>>> print(instance.introspect())
- - Data():
0 \x00 length: 1 (UBInt16)
1 \x01
- - data: String():
2 \x61 u'a'
>>> instance.length.set(16)
>>> print(instance.introspect())
- - Data():
0 \x00 length: 16 (UBInt16)
1 \x10
- - data: String():
2 \x61 u'a'
3 \x62 u'b'
4 \x63 u'c'
5 \x64 u'd'
6 \x65 u'e'
7 \x66 u'f'
8 \x67 u'g'
9 \x68 u'h'
10 \x69 u'i'
11 \x6a u'j'
12 \x6b u'k'
13 \x6c u'l'
14 \x6d u'm'
15 \x6e u'n'
16 \x6f u'o'
17 \x70 u'p'
The length
of the oser.String
is a callable
here that returns
the length
’s value.
For simple true-false-decisions oser.IfElse
can be used.
>>> class IfElseData(ByteStruct):
... def __init__(self):
... super(IfElseData, self).__init__()
... self.true_false = UBInt8(1)
... self.data = IfElse(condition=lambda self: bool(self.true_false.get()),
... if_true=UBInt8(1),
... if_false=UBInt32(0xffffffff)
... )
...
>>> instance = IfElseData()
>>> print(instance.introspect())
- - IfElseData():
0 \x01 true_false: 1 (UBInt8)
1 \x01 data: 1 (UBInt8)
>>> instance.true_false.set(0)
>>> print(instance.introspect())
- - IfElseData():
0 \x00 true_false: 0 (UBInt8)
1 \xff data: 4294967295 (UBInt32)
2 \xff
3 \xff
4 \xff
The condition
of oser.IfElse
is a callable
here that returns
the true_false
’s value as bool
.
A more complex example is a oser.Switch
that decides
the type of the payload.
>>> class SwitchData(ByteStruct):
... def __init__(self):
... super(SwitchData, self).__init__()
... self.type = UBInt8(1)
... self.data = Switch(condition=lambda self: self.type.get(),
... values={
... 1: UBInt8(1),
... 2: UBInt16(2),
... 4: UBInt32(3)
... },
... default=Null())
...
>>> instance = SwitchData()
>>> print(instance.introspect())
- - SwitchData():
0 \x01 type: 1 (UBInt8)
1 \x01 data: 1 (UBInt8)
>>> instance.type.set(2)
>>> print(instance.introspect())
- - SwitchData():
0 \x02 type: 2 (UBInt8)
1 \x00 data: 2 (UBInt16)
2 \x02
>>> instance.type.set(4)
>>> print(instance.introspect())
- - SwitchData():
0 \x04 type: 4 (UBInt8)
1 \x00 data: 3 (UBInt32)
2 \x00
3 \x00
4 \x03
>>> instance.type.set(3)
>>> print(instance.introspect())
- - SwitchData():
0 \x03 type: 3 (UBInt8)
- - data: Null
The condition
of oser.Switch
is a callable
here that returns
the type
’s value. If type
is not in [1,2,4]
oser.Null
is used.
To access parent elements use oser.OserNode.up()
.
>>> class SubSwitch(ByteStruct):
... def __init__(self):
... super(SubSwitch, self).__init__()
... self.data = Switch(condition=lambda self: self.up().type.get(),
... values={
... 1: UBInt8(1),
... 2: UBInt16(2),
... 4: UBInt32(3)
... },
... default=Null())
...
>>>
>>> class SwitchData2(ByteStruct):
... def __init__(self):
... super(SwitchData2, self).__init__()
... self.type = UBInt8(1)
... self.data = SubSwitch()
...
>>> instance = SwitchData()
>>> print(instance.introspect())
- - SwitchData():
0 \x01 type: 1 (UBInt8)
1 \x01 data: 1 (UBInt8)
>>> instance.type.set(2)
>>> print(instance.introspect())
- - SwitchData():
0 \x02 type: 2 (UBInt8)
1 \x00 data: 2 (UBInt16)
2 \x02
>>> instance.type.set(4)
>>> print(instance.introspect())
- - SwitchData():
0 \x04 type: 4 (UBInt8)
1 \x00 data: 3 (UBInt32)
2 \x00
3 \x00
4 \x03
>>> instance.type.set(3)
>>> print(instance.introspect())
- - SwitchData():
0 \x03 type: 3 (UBInt8)
- - data: Null
In this example the type
is accessed by self.up().type.get()
.
It is also possible to create variable length repeated fields using
oser.Array
.
>>> class VariableLengthArray(ByteStruct):
... def __init__(self):
... super(VariableLengthArray, self).__init__()
... self.length = UBInt16(1)
... self.data = Array(
... length=lambda self: self.length.get(),
... prototype=UBInt8,
... values=[UBInt8(ii) for ii in range(256)])
...
>>> instance = VariableLengthArray()
>>> print(instance.introspect())
- - VariableLengthArray():
0 \x00 length: 1 (UBInt16)
1 \x01
- - data: Array():
- - [
2 \x00 @0: 0 (UBInt8)
- - ]
>>> instance.length.set(5)
>>> print(instance.introspect())
- - VariableLengthArray():
0 \x00 length: 5 (UBInt16)
1 \x05
- - data: Array():
- - [
2 \x00 @0: 0 (UBInt8)
3 \x01 @1: 1 (UBInt8)
4 \x02 @2: 2 (UBInt8)
5 \x03 @3: 3 (UBInt8)
6 \x04 @4: 4 (UBInt8)
- - ]
In this example length
is a callable
like in the variable
length string example.
Copying¶
OSER instances can be deeply copied using copy.deepcopy()
.
Shallow copies are not allowed since the context of each element must be distinct.
If copy.copy()
is applied on an OSER instance and exception is raised.
While being copied encode()
is called with all its side effects.
>>> from __future__ import absolute_import, division, print_function, unicode_literals
>>>
>>> import oser
>>> import copy
>>>
>>>
>>> class Struct(oser.ByteStruct):
... def __init__(self):
... super(Struct, self).__init__()
... self.a = oser.ULInt8(1)
...
>>> s = Struct()
>>> id(s)
6596088
>>> print(s)
Struct():
a: 1 (ULInt8)
>>>
>>> deep_copy = copy.deepcopy(s)
>>> id(deep_copy)
38738520
>>>
>>> s.a.set(100) # this does not influence the copy
>>> print(deep_copy)
Struct():
a: 1 (ULInt8)