Datasources

The @electricui/core-timeseries library provides powerful data manipulation functionality used to format inbound data for logging, buffer historical data for graphs, and build maths helpers for UI-side calculations.

What is a DataSource?

DataSources provide a unified interface for handling timeseries events used in charting and logging systems. They are not limited to streams of data from hardware, though that is their most common use case.

DataSources provide their own persistent storage used to query historical data.

The most common DataSource is a MessageDataSource driven by a message identifier. It produces events when specified messages arrive. We get the MessageDataSource with a React Hook which goes in your Component, before return().

import { useMessageDataSource } from '@electricui/core-timeseries'
 
const OverviewPage = () => {
  const speedSource = useMessageDataSource('speed')
 
  return (
    <React.Fragment>
      UI here
    </React.Fragment>
  )
}

LineChart, Console and PolledCSVLogger are some of the components which accept DataSources for time-series variable streams.

How it works

MessageDataSource is fed events by the QueryableMessageIDProvider which runs as part of the transport manager.

Additionally, the QueryableMessageIDProvider provides a method allowing the developer to manually process the incoming Message into an Event suitable for the timeseries system.

The default processor behaviour would like this (simplified for a example structure):

// @filename: template/src/transport-manager/index.tsx
import {
  Event,
  QueryableMessageIDProvider,
  ElectronIPCRemoteQueryExecutor,
} from '@electricui/core-timeseries'
import { Message } from '@electricui/core'
import { timing } from '@electricui/timing'
import { deviceManager } from './config'
 
const remoteQueryExecutor = new ElectronIPCRemoteQueryExecutor()
const queryableMessageIDProvider = new QueryableMessageIDProvider(
  deviceManager,
  remoteQueryExecutor,
)
 
queryableMessageIDProvider.setCustomProcessor('speed', (message, emit) => {
  // Build the event at the current time
  const event = new Event(message.metadata.timestamp, {
    gps: message.payload.gps,
    wheels: message.payload.wheels,
  })
 
  // Emit the event
  emit(event)
})

Working with custom types

For projects using custom-typed data, the DataSource is able to infer the type correctly.

const speedDSTypeInfered = useMessageDataSource("speed")
              
const speedDSTypeInfered: MessageDataSource<SpeedSensors, "speed">

In practice, this allows for useful code-completion hints, and errors if an invalid accessor is written.

export type SpeedSensors = {
  gps: number
  wheels: number
}
 
const OverviewPage = () => {
  const speedDS = useMessageDataSource(MSGID.SPEED)
  return (
    <React.Fragment>
      <ChartContainer>
        <LineChart dataSource={speedDS} accessor={event => event.radar}/>
Property 'radar' does not exist on type 'SpeedSensors'.2339Property 'radar' does not exist on type 'SpeedSensors'.
        <RealTimeDomain window={10000} />
      </ChartContainer>
    </React.Fragment>
  )
}

If you're interested in how this works, read the deep-dive writeup on our blog!

If the type needs to be enforced manually, specify it globally with the TypedState file and use it as normal.

// /src/application/typedState.ts
export enum MSGID {
  SPEED = 'speed',
}
 
export type SpeedSensors = {
  gps: number
  wheels: number
}
 
declare global {
  interface ElectricUIDeveloperState {
    speed: SpeedSensors
  }
}

const speedDSTypeEnforced = useMessageDataSource(MSGID.SPEED)
              
const speedDSTypeEnforced: MessageDataSource<SpeedSensors, MSGID>

Retiming Events

By default, Events are timestamped by the transport manager.

In situations where a large amount of data is streaming from hardware, it is possible that dropped packets, a busy transport, or heavily loaded transport manager may not process data immediately as it arrives.

If this occurs, charts may show visual errors due to out-of-order packets or erroneous delays. The recommended approach in these circumstances is to provide hardware timing information in the packet.

// template/src/transport-manager/index.tsx
 
interface SpeedPayload {
  received: number // timestamp in ms since epoch, provided by hardware
  sensor: number // adc readings
}
 
queryableMessageIDProvider.setCustomProcessor(
  'speed',
  (message: Message<SpeedPayload>, emit) => {
    // the epoch needs to be subtracted to convert the global time provided by hardware to session time
    const epoch = timing.timeOrigin
 
    // Custom time from the message
    const event = new Event(
      message.payload!.received - epoch,
      message.payload!.sensor,
    )
    emit(event)
  },
)

It is also significantly more efficient to batch multiple events in the same packet.

By emitting multiple events for an inbound message, the downstream consumers can't tell a difference from a standard DataSource.

// template/src/transport-manager/index.tsx
 
interface PackedPayload {
  delta: number // the delta time between each subsequent event
  values: number[] // the adc values of each event
}
 
queryableMessageIDProvider.setCustomProcessor(
  'speed',
  (message: Message<PackedPayload>, emit) => {
    const start = message.metadata.timestamp
 
    // Each packet has 512 sensor readings
    for (let index = 0; index < 512; index++) {
      // Create the timestamp for this value, and then push the retimed event
      const time = start + message.payload!.delta * index
      const event = new Event(time, message.payload!.values[index])
      emit(event)
    }
  },
)

Getting the current timestamp

In certain situations, knowing the timestamp for 'now' in the datasource or transformer can be useful, rather than the included packet timestamp.

The @electricui/timing package provides the timing.now() function, returning the current time in milliseconds since session start, and the timing.timeOrigin constant, storing the unix time in milliseconds since unix epoch of when the session began.

Timing is stored as floating point milliseconds since the start of the 'session' within the timeseries ecosystem.

Global time can be converted to session time using the timing.timeOrigin constant.

import { timing } from '@electricui/timing'
 
const timestamp = timing.now()
 
// timestamp of the start of session in milliseconds since unix epoch
const epoch = timing.timeOrigin 

Manipulating Data

Using a DataSource in isolation is fine when plotting raw signals streamed from hardware, but when 'client-side' signal processing or reformatting is required, custom transport-manager processors aren't suitable.

DataTransformers help you guide the data through a transformation pipeline with minimal loss of performance, whilst maintaing modular code.

import { leakyIntegrator } from '@electricui/dataflow'
import { useDataTransformer } from '@electricui/timeseries-react'
import { useMessageDataSource } from '@electricui/core-timeseries'
 
const OverviewPage = () => {
  const sensorDataSource = useMessageDataSource('adc')
 
  const filteredDS = useDataTransformer(() => {
    return leakyIntegrator(sensorDataSource, 4)
  })
 
  return (
    <React.Fragment>
      Chart of the averaged data
    </React.Fragment>
  )
}

The full guide with a range of worked examples is recommended reading:

Decimation

It is usually unnecessary to display more points than there are pixels on the screen.

To improve performance by default, all MessageDataSources have a "time stride" based decimator attached.

This time stride decimator buckets the data into 4096 sections, and fetches the first Event within each bucket. This gives similar performance whether ten thousand or ten million events are stored in the persistence engine.

The decimator can be modified using the setDefaultDecimatorFactory method to change the default for all MessageIDs, or the setCustomDecimatorFactory method for changing a specific MessageID.

import { timeStrideDecimationFromQuery } from '@electricui/dataflow'
 
// /src/transport-manager/index.tsx
 
queryableMessageIDProvider.setDefaultDecimatorFactory(dataSource => {
  // By default, don't do any decimation
  return dataSource
})
 
queryableMessageIDProvider.setCustomDecimatorFactory('speed', dataSource => {
  // But for the `speed` messageID, do the regular decimation
  return timeStrideDecimationFromQuery(dataSource)
})

Any DataFlow can be used as a decimator, so if the time stride decimator doesn't suit your needs, a custom one can be built.

Persisting DataSources

Standalone, a DataSource only provides a stream of events. However, when a user leaves a page and eventually returns there's an implicit expectation that the graphed data will show the hardware's state while they were gone.

To achieve this, and the more complex functionality which powers decimation and the more complicated transformers, we persist the last 10k events with an in memory database.

Electric UI is configured with a balanced default setup, but we describe implementation details and how to customise the store in the Persistence guide.