Advanced data handling techniques

Structures underpin the representation of custom data types and provide logical grouping and syntactical sugar to developers. Structures or custom types allow developers to pass more complex data between functions and semantically improve variable access with dot notation. Use of these language features is a vital part of serious software development projects.

Electric UI encourages the use of custom types to pipeline related variables into one packet, and reducing the number of message ID's in use.

Microcontroller Side

If you are using the electricui-embedded C library and protocol, connecting an existing structure to the UI is straight-forward1.

This C example snippet is based on the hello-blink example. We define and declare a structure variable, and change the tracked variables to use a EUI_CUSTOM macro with the new variable. We'll call it led.

  1. typedef struct {
  2. uint16_t glow_time;
  3. uint8_t enable;
  4. } blinker_t;
  5. blinker_t led_settings = { 350, 1 };
  6. uint8_t led_state = 0;
  7. eui_message_t tracked_variables[] =
  8. {
  9. EUI_CUSTOM( "led", led_settings ),
  10. EUI_UINT8( "led_state", led_state ),
  11. };

In the main loop, some little changes are needed to use the new structure members instead of discrete variables.

  1. void loop()
  2. {
  3. serial_rx_handler();
  4. if( led_settings.enable )
  5. {
  6. if( millis() - led_timer >= led_settings.glow_time )
  7. {
  8. led_state = !led_state;
  9. led_timer = millis();
  10. eui_send_tracked("led_state");
  11. }
  12. }
  13. digitalWrite( LED_BUILTIN, led_state );
  14. }

After flashing this example to hardware (and using the default UI template as reference), you'll notice that the slider control don't work anymore.

When sending and receiving variables with a custom type, the design of Electric UI assumes that the UI has suitable encoders and decoders to handle unpacking the bytes matching the memory model of the microcontroller.

For tips on how to most effectively use structures with C, we recommend reading this article on structure padding optimisation techniques to get the most bang for your byte.

Structure safety advisory

In situations where exposing your structure memory is like holding a gun to your foot, we strongly suggest pointing Electric UI to a buffer of bytes which are safe to serialize against over the project's lifecycle.

You'll want to do this when your project:

  • Wants portability between microcontroller architectures and/or compilers,
  • The compiler doesn't guarantee structure repacking or padding behaviour,
  • You are using a higher level construct to declare your data structure in abstract memory (traits, C++ vector),
  • Uses a language which doesn't respect developer structure ordering, such as Rust without repr(C),
  • You want to be explicit, at the expense of complexity (and a bit of overhead).

Data Structure Alignment

One small catch when working with structures which differ from the above example; it's quite likely that your microcontroller and compiler doesn't actually shove your structured data hard against each element due to word boundaries.

Consider the following example:

  1. typedef struct {
  2. uint8_t a;
  3. uint16_t b;
  4. int8_t c;
  5. float d;
  6. } sensor_settings_t;

We would probably think that the bytes of b start right after our a variable, and therefore our structure would be nicely represented in memory as written, as shown below.

b0b1b2b3b4b5b6b7
ABBCDDDD

Due to the architectural design of many processors its more efficient to read and write memory in multiples of bytes. As a result, the compiler inserts padding to ensure that b starts on the word boundary, and then another is added to ensure d starts with correct alignment. A 2-byte (16-bit) alignment was used for this example, which results in 2 additional padding bytes, but 4-byte (32-bit) alignment is more common with the popularity of modern 32-bit microcontrollers.

b0b1b2b3b4b5b6b7b8b9
A-BBC-DDDD

Because the structure has two smaller members which can fit neatly inside the boundary, manually re-ordering the structure by moving c forwards lets the compiler align the larger types of b and d without padding, reducing the memory impact and transfer overhead slightly.

b0b1b2b3b4b5b6b7
ACBBDDDD

Not all structures can be entirely free of padding (without special help from the compiler), and in these cases the UI side codec just needs to be aware of these additional bytes, and you need to be aware of portability differences if running your embedded code on a mix of architectures.

Forced structure packing

This feature isn't supported by all compilers or platforms, and if you are running a non-C style language it might not be possible at all.

GCC can be prevented from adding padding data to structures with an explicit attribute, __attribute__((__packed__)), which would look something like this

  1. typedef struct {
  2. uint8_t a;
  3. uint16_t b;
  4. int8_t c;
  5. float d;
  6. } __attribute__((__packed__)) sensor_settings_t;

This would result in a data structure which contains no padding. If we wanted to, we could also request the particular padding alignment with the following tweak (2-byte alignment):

  1. typedef struct {
  2. uint8_t a;
  3. uint16_t b;
  4. int8_t c;
  5. float d;
  6. } __attribute__(( packed, aligned(2) )) sensor_settings_t;

Some compilers use variations, like #pragma pack(n) to suppress or select the number of padding bytes. For our 32-bit architecture, this gives an un-padded structure in memory at the potential expense of memory access speed3. Neat!

This approach isn't immediately recommended due to the compiler specific nature of these features and the rare incompatibilities that might occur on certain platforms2. Structure alignment settings are somewhat up to compiler implementation so be wary.

If you are using a resource constrained microcontroller and can't reorder the structure or manually manage a block of memory, this might be one of the optimisations to look into.

User Interface Codecs

With our structure setup on the embedded side, we need to teach the UI how to read inbound packets and correctly form data into an outbound packet.

Codecs consist of a filter to catch relevant packets, a decoder to convert the inbound bytestream into a Typescript object, and an encoder to convert a Typescript object back into a buffer of bytes for hardware.

The default template generated by arc has a demonstration codec setup, so open the src/transport-manager/config/codec.tsx file.

Setting up a Codec

In codecs.ts we import the required libraries, create a new class LEDCodec, filter messages with an ID of led, and then build out the encoder and decoder logic.

Encoders and decoders are pure functions that convert an object to and from a Buffer.

The example follows best-practices and defines a LEDSettings type to hold the UI side data, as strictly typed objects allow Electric UI to generate 'compile time errors' if misused in the frontend.

import { Codec } from '@electricui/core'
import { SmartBuffer } from 'smart-buffer'
  1. export type LEDSettings = {
  2. glowTime: number
  3. enable: number
  4. }
  5. export class LEDCodec extends Codec<LEDSettings> {
  6. filter(message: Message): boolean {
  7. return message.messageID === 'led'
  8. }
  9. encode(payload: LEDSettings): Buffer {
  10. // ...
  11. }
  12. decode(payload: Buffer): LEDSettings {
  13. // ...
  14. }
  15. }
  16. // Create the instances of the codecs
  17. export const customCodecs = [
  18. new LEDCodec(),
  19. ]

A codec needs both encode() and decode() methods. In situations where a microcontroller just provides read-only sensor data the encoder should throw an error.

  1. export class ReadOnlyCodec extends Codec<LEDSettings> {
  2. filter(message: Message): boolean {
  3. return message.messageID === 'led'
  4. }
  5. encode(payload: LEDSettings): never {
  6. throw new Error("The led messageID is read only.")
  7. }
  8. decode(payload: Buffer): LEDSettings {
  9. // ...
  10. }
  11. }

Codecs are added as part of the communications pipeline for each transport in use. In serial.ts, you'll see a block of code which creates the pipeline, adds the default codecs (used for integer/float/callback etc), and then adds the codecs with .addCodecs()as shown:

import { customCodecs } from './codecs'
  1. // Halfway down, inside the serialTransportFactory
  2. const codecPipeline = new CodecDuplexPipeline()
  3. codecPipeline.addCodecs(customCodecs)
  4. codecPipeline.addCodecs(defaultCodecList)

This approach can be copied to add codecs spanning other files, just create a complexCodec.tsx file, then import it and add it to the pipeline of your transports.

Decoding

The payload argument is a NodeJS Buffer. It can be read directly, but for most use cases a helper library called SmartBuffer is recommended.SmartBuffer provides a simple but powerful approach to manipulating buffers of bytes, as it automatically tracks the cursor position as the raw bytes are read or written.

Little Endian and Big Endian functions are available for n-byte types (LE is most typical with microcontrollers). These functions accept an optional offset argument which moves the start address of the decode in the buffer, useful for jumping over padding bytes.

  1. export class LEDCodec extends Codec<LEDSettings> {
  2. decode(payload: Buffer): LEDSettings {
  3. const reader = SmartBuffer.fromBuffer(payload)
  4. // Read the glow time first,
  5. const glowTime = reader.readUInt16LE()
  6. // then read the padding byte out to automatically increment the cursor
  7. reader.readUInt8()
  8. // Read out the enable boolean
  9. const enable = reader.readUInt8()
  10. // Build our object
  11. const settings = {
  12. glowTime,
  13. enable,
  14. }
  15. // Return the completed object, the type checking will
  16. // validate that it matches the type of LEDSettings
  17. return settings
  18. }
  19. }

The SmartBuffer methods follow the stdtype.h set used in electricui-embedded.

  1. a: reader.readUInt8(),
  2. c: reader.readInt8(),
  3. b: reader.readUInt16LE(),
  4. d: reader.readFloatLE(),

If you want to access data in the buffer manually, just ignore SmartBuffer and access the payload argument directly.

Note the methods available are different, it doesn't support reading strings for example, there is no automatic reading cursor. However for reading packed number types it can be sufficient. Below is a reimplementation of the above.

  1. export class LEDCodec extends Codec<LEDSettings> {
  2. // ... filter, encoder
  3. decode(payload: Buffer): LEDSettings {
  4. const settings = {
  5. glowTime: payload.readUInt16LE(0), // the offset must be passed as an argument
  6. enable: payload.readUInt8(2), // the offset must be passed as an argument
  7. }
  8. return settings
  9. }
  10. }

Encoding

The encoder works in reverse to the decoder implementation. SmartBuffer again has utility in automatically tracking the cursor.

  1. export class LEDCodec extends Codec<LEDSettings> {
  2. // ... filter
  3. encode(payload: LEDSettings): Buffer {
  4. // SmartBuffers automatically keeps track of read and write offsets / cursors.
  5. const packet = new SmartBuffer({ size: 4 })
  6. packet.writeUInt16LE(payload.glow_time)
  7. packet.writeUInt8(0x00) // Write a padding byte.
  8. packet.writeUInt8(payload.enable)
  9. // Push it up the pipeline
  10. return packet.toBuffer()
  11. }
  12. // ... decoder
  13. }

The null case is handled automatically if importing the default codecs. Any outgoing message with a payload of null will have a zero length payload on the hardware side. Any incoming message with a zero length payload will have a null payload on the UI side.

Inline integer-enum translation

Often firmware developers will express operating states as an integer value and enum.

Typescript enums allow similar functionality, referencing an integer state by an enum. As a simple demonstration of syntax:

  1. export enum TEMPERATURE_ERROR {
  2. NONE = 0,
  3. NOT_CONNECTED,
  4. LOW,
  5. HIGH,
  6. GENERIC,
  7. }
  8. // Usage
  9. const none = TEMPERATURE_ERROR.NONE // 0
  10. const high = TEMPERATURE_ERROR.HIGH // 3

The codec can simply pass the integer through to the UI element. The redux devtools and internal representation will be the integer, however the source code can reference the named member.

Note that this approach requires the integer value on the embedded side to match the integer value of the enum on the UI side.

  1. export interface SensorMessage {
  2. sensorError: TEMPERATURE_ERROR
  3. temperature: number
  4. }
  5. export class SensorCodec extends Codec<SensorMessage> {
  6. // ... filter
  7. encode(payload: SensorMessage): Buffer {
  8. const packet = new SmartBuffer()
  9. packet.writeUInt8(message.payload.sensor_error)
  10. packet.writeUInt8(message.payload.temperature)
  11. return packet.toBuffer()
  12. }
  13. decode(payload: Buffer): SensorMessage {
  14. const reader = SmartBuffer.fromBuffer(message.payload)
  15. return {
  16. sensorError: reader.readUInt8(),
  17. temperature: reader.readuint8(),
  18. }
  19. }
  20. }

When interacting with the enum in the user interface code, import the enum definition, and use it in the same manner.

  1. import {
  2. TEMPERATURE_ERROR,
  3. } from './../../transport-manager/config/codecs'
  4. // Function which returns nicely formatted text based on sensor state
  5. const PrettyTempErrorPrinter = () => {
  6. const temp_state = useHardwareState(state => state.temp.sensorError)
  7. let error_text: string = 'Invalid'
  8. if (temp_state === TEMPERATURE_ERROR.NONE) {
  9. error_text = "Everything is OK!"
  10. } else if (temp_state === TEMPERATURE_ERROR.NOT_CONNECTED) {
  11. error_text = "Please connect the temperature probe"
  12. } else {
  13. error_text = "Warning. Temperature reading error."
  14. }
  15. return <span>{error_text}</span>
  16. }

Manipulating data in codecs

In some situations, you may be sending data to/from the UI which needs to be converted into a more human friendly format. While this can be done at the presentation layer, its also possible to manipulate the data at the codec side. Codec transformations are then consistent across all UI components reducing the amount of frontend boilerplate.

This approach can be abused to minimise data transfer, useful in larger arrays of data or in constrained bandwidth applications.

In this example, we'll convert a temperature value which is being transmitted as a single unsigned byte. In this case, we know the temperature conversion range falls within a -20°C to +60°C range.

( temp°C + 10 ) * 4

The general flow of data might look like this:

Value
Temperature Sensor2.60V
ADC Sample768 counts
Steinhart temperature conversion48.75°C
Integer value sent to UI48
Transformed value sent to UI235

By offsetting the temperature value and multiplying it, we lose less precision from our usable range compared to using a standard uint8. There is nearly zero cost to performing this kind of transform, but results in more detailed data, drastically improving the appearance of a graph for the same bandwidth.

Integer Value of 48Transformed Value of 235
1°C steps0.25°C steps
Positive temperatures onlySupports subzero
Doesn't use most of the 8-bit rangeMaximises usage of the 8-bit range
Human readableNeeds transformation before display

In the decoder, just perform maths as you would normally.

  1. export interface MotorSettings {
  2. micro: number
  3. motor: number
  4. external: number
  5. }
  6. export class MotorCodec extends Codec<MotorSettings> {
  7. // ... filter, encoder
  8. decode(payload: Buffer): MotorSettings {
  9. const reader = SmartBuffer.fromBuffer(message.payload)
  10. return {
  11. micro: (reader.readUInt8() / 4) - 10,
  12. motor: (reader.readUInt8() / 4) - 10,
  13. external: (reader.readUInt8() / 4) - 10,
  14. }
  15. }

This same style of transform can occur in an encoder to transform a user configurable value back into the desired form for the microcontroller.

While this example is simplified, this kind of transform could be used to convert a RGB colour structure into HSV, transform a 8-bit PWM value to a 0-100% range duty cycle, or create an new watts member from the incoming volts and current structure elements.

Using structure elements with UI components

When using a structured variable with components we use standard accessor syntax.

  1. // You can poll the with normal messageID string syntax
  2. <IntervalRequester
  3. variables={['led_state', 'led']}
  4. interval={250}
  5. />
  6. <Printer accessor="lit_time" />
  7. // becomes
  8. <Printer accessor={ state => state.led.glow_time } />
  9. <Switch
  10. unchecked={0}
  11. checked={1}
  12. accessor="led_blink"
  13. writer={(state, value) => (state.led_blink = value)}
  14. />
  15. // uses the same syntax as any component using writer props,
  16. <Switch
  17. unchecked={0}
  18. checked={1}
  19. accessor={state => state.led.enable}
  20. writer={(state, value) => (state.led.enable = value)}
  21. />

The slider has a little more syntax, as a slider supports multiple handles to mutate different structure members.

  1. <Slider>
  2. <Slider.Handle accessor="lit_time" />
  3. </Slider>
  4. // becomes
  5. <Slider
  6. writer={(state, values) => {
  7. state.led.glow_time = values.duration
  8. }}
  9. >
  10. <Slider.Handle name="duration" accessor={state => state.led.glow_time} />
  11. </Slider>

Bitfields

Bitfields are commonly used on embedded platforms to pack multiple settings into a single byte, saving space, or to load into a peripheral's register.

Microcontroller

Microcontroller bit-fields behave the same way as structures with manually specified type sizing.

You can specify the size of each member in C by using the following syntax:

  1. struct {
  2. // Members are 1-bit in size
  3. unsigned alarm_on : 1;
  4. unsigned light_on : 1;
  5. } UserSettings;

This is also commonly achieved using bit-masks to manually mutate the individual bits of a byte-typed variable.

Codecs and UI usage

Below is how you have a bitfield of booleans.

import { Codec } from '@electricui/core'
import {
Pack,
Unpack,
BitFieldObject,
} from '@electricui/utility-bitfields'
  1. export interface BitFieldSettings extends BitFieldObject {
  2. bitA: boolean
  3. bitB: boolean
  4. bitC: boolean
  5. bitD: boolean
  6. bitE: boolean
  7. bitF: boolean
  8. bitG: boolean
  9. bitH: boolean
  10. }
  11. const bitFieldOrder = [
  12. 'bitA', // 0bX0000000
  13. 'bitB', // 0b0X000000
  14. 'bitC', // 0b00X00000
  15. 'bitD', // 0b000X0000
  16. 'bitE', // 0b0000X000
  17. 'bitF', // 0b00000X00
  18. 'bitG', // 0b000000X0
  19. 'bitH', // 0b0000000X
  20. ]
  21. export class BitfieldCodec extends Codec<BitFieldSettings> {
  22. filter(message: Message): boolean {
  23. return message.messageID === 'bitfield'
  24. }
  25. encode(payload: BitFieldSettings): Buffer {
  26. return Pack(payload, bitFieldOrder)
  27. }
  28. decode(payload: Buffer): BitFieldSettings {
  29. // otherwise it's just a string, pass it on as is.
  30. return Unpack(payload, bitFieldOrder)
  31. }
  32. }

Arrays

Arrays are easy to work with!

On the embedded side (assuming electricui-embedded) the size of the data structure is silently grabbed as part of the macros used in the eui_message_t array.

This means for any default typed array, the UI automatically handles the sizing and dissects the variable into elements.

For custom types, we recommend using SmartBuffer in a loop to collect each items of the array.

To use a specific element from the array in the UI

<Printer accessor={state => state.leds[3].glow_time} />

  1. Please read the languages and structure ordering safety note in this guide
  2. ARM M0 cores are stricter about alignment than more recent ARM variants, which have less or no impact with poorly aligned data access.
  3. Strict alignment architectures like SPARC don't allow it at all