ElectricUI Binary Protocol

The default serial protocol for Electric UI provides a fast way to get started.

  • Supports human readable message identifiers of up to 15 characters,
  • Payload sizes up to 1kB per packet,
  • Inbuilt support for splitting larger data structures over multiple packets,
  • Included type information allows for zero-configuration,
  • Features isolated namespaces for developer messages and internal packets,
  • Supports acknowledgement numbers and query behaviour,
  • Bandwidth overhead of 7-9 bytes per packet1
  • Checksum and frame encoding provide error detection and fast failure recovery.

Anatomy of a packet

Packets consist of binary data and have a header, message identifier (ASCII, hash or index based), optional offset metadata for multi-packet payloads, payload data, and a checksum.

Packet layout has Header, Message ID, Offset, Payload, Checksum

The packets are then encoded using the COBS framing algorithm.

A packet is wrapped with COBS encoding

Endianness

As the protocol is designed for interoperability with microcontrollers and modern systems, data is in the Little Endian format (LE). We don't use Network Order because these packets aren't subject to historical standards.

If you are using a big endian system, we suggest the recommended reading here \$TODO.

The header stores feature flags, sizing and type information, and uses 3-bytes of space. Feature bitflags operate independent of each other.

DataBits AllocatedDescription
Payload Length10Size in bytes up to 1024 bytes. >0 lengths implies intent to write data.
Payload Type4
Message ID Length4Size in bytes up to 15
Internal Message Flag1High when packet is for internal use (outside the developer's scope)
Offset Payload Flag1High when packet uses the payload offset word.
Response Flag1When high, a response with the current value of the target variable is returned.
Sequence Number3For non-zero sequence numbers, we expect an acknowledgement with the same sequence number.

The packing of this data into the header bytes minimises effects of word alignment padding.

Bytes123
Bits123456781234567812345678
DataPayload LengthTypeInternalOffsetID LengthQuerySequence

Writing data (payload length > 0) *and* query should trigger a response with the new data, not the previous data.

Types

The payload type field allows for automatic conversions on the UI side. These type values complement stdint.hs standard type schemes ( uint8_t, int32_t etc ).

IndexTypeBytesDescriptionRange
0Callback0Used to trigger a function call or eventn/a
1Customn/aFollows data structure's memory modelLimitless!
2Offset Metadatan/a
3Byte1
4Character1Character (intended to be printable). Character arrays are strings
5int81Signed 8-bit integer±127
6uint81Unsigned 8-bit integer+255
7int162Signed 16-bit integer±32,767
8uint162Unsigned 16-bit integer+65,535
9int324Signed 32-bit integer±2,147,483,647
10uint324Unsigned 32-bit integer+4,294,967,295
11float4Standard precision floating point value±3.4E+38
12double8Double precision floating point±1.7E+308

64-bit integers aren't commonly used on embedded systems, so we recommend using a custom type which allows for more intuitive handling on the UI side.

Message Identifier

The message identifier (often abbreviated as msgID) is a variable number of bytes which denote the transmitted variable.

The default implementation suggests a ASCII formatted string with <=15 character length. This scheme allows for human-readable nmemonic identifiers on the embedded and UI side, aiding in usability and debugging, while allowing easy referencing in the application layer.

The protocol does force 1-byte ID minimum lengths, however there is no restriction on the data. Non-printable characters are legal, and simple modifications to the ID system allow index based identifiers to save precious bytes on constrained or production systems.

While the protocol technically supports a far larger2 number of unique messageID's than recommended, your design should use no more than a dozen or two (more suggests architectural issues, read the Custom Types documentation).

Offset (optional)

Truthy offset bit values enable a 16-bit address offset from the base address of the variable. This adds support for large data structures and arrays, even with a small packet size.

Payload

Payload data formatting matches the native device representations, little-endian formatted data structure held in memory.

This allows for low-cost output of the data and a simple direct memory copy from the valid inbound packet to the variable.

Checksum

A 2-byte checksum word is at the end of each packet. We use the CRC16-CCITT-FALSE checksum with a seed word of 0xFFFF, processed on the embedded side as data streams in/out.

One notable aspect of cyclic redundancy check algorithms is ensuring that the common failure modes will not trigger with real-world data, and the collision space is sufficiently large to reasonably avoid false-positives. The tradeoff between implementation overhead (bandwidth, memory, code space) favours lower-end microcontrollers.

Of particular interest with our selected CRC implementation, it will self-nullify when presented with bytes matching the current working CRC value.

  • We seed with 0xFFFF,
  • If the first bytes processed are also 0xFFFF, the output CRC will be 0x0000,
  • when followed by at least 2 successive 0x00 bytes, the CRC will not be unique while the chain of 0x00 bytes continues.

This is not an issue for our packet layout, as the header doesn't allow a series of problematic bytes to occur due to the natural ordering of bits.

Encoding scheme

We take the entire packet described above (header through to checksum) and encode it using the lightweight Consistent Overhead Byte Stuffing (COBS) algorithm.

The operating theory is rather simple, for any value 0x00 in our packet, we replace it with an offset value describing the number of bytes we expect to count before the next 0x00 byte.

Because we no longer have any 0x00 bytes in our data, we can use the 0x00 byte as our reserved delimiter character, which provides frame synchronisation.

DIAGRAM HERE

This approach ensures the overhead is one byte per 255 bytes, and allows the parsing statemachines to instantly reset if any of the reserved 0x00 bytes appear. Failing fast means we lose, at most, the one packet if errors cause erraneous behaviour.

DIAGRAM/EXAMPLES OF HOW THIS SAVES US

Considerations

While developing a solid and reliable protocol has been a priority, it does not provide guarantees for information sensitive or safety critical applications. We strongly recommend finding or developing a suitable protocol for applications where message reliability, timing, error identification or cryptographic safety is required.

Protocol Functionality

While the underlying protocol is basic, the UI uses internally namespaced messages to provide connection monitoring, handshakes and fault recovery.

Handshaking and Device Discovery

Automatic retries and error handling

Offset Packets

Electric UI's default protocol supports up to 1kB payload transfers per packet, though the default payload length is smaller to help keep the inbound buffer slim.

1kB packets don't go far with images or buffered streams of sensor data, and we don't want to send arrays between the microcontroller and PC element-by-element either!

Electric UI has an automatic solution to splitting larger data structures into multiple messages. By providing a 16-bit payload offset value we are able to send, recieve and query smaller parts of a larger block of memory.

Diagram of multiple offset messages (long bitstream with arrows pointing to packets and explaining the offset)

The UI maintains a copy of the data structure, and will fill sections in as required. Partial updates allow the UI to specifically write to a given position inside the variable's memory.

This bandwidth reduction technique is automatically used when UI actions trigger a write of the large variable.


  1. Developer configurable settings can tweak underlying overheads.
  2. 25515 uniques, or about an undecillion, about 200 million of which are printable