Datasources

The @electricui/core-timeseries library provides powerful data manipulation functionality used to format inbound data for logging, buffer historical data for graphs, and build maths helpers for UI-side calculations.

What is a DataSource?

DataSources provide a unified interface for handling timeseries events used in charting and logging systems. They are not limited to streams of data from hardware, though that is their most common use case.

DataSources provide their own persistent storage used to query historical data.

The most common DataSource is a MessageDataSource driven by a message identifier. It produces events when specified messages arrive. We get the MessageDataSource with a React Hook which goes in your Component, before return().

import { useMessageDataSource } from '@electricui/core-timeseries'
 
const OverviewPage = () => {
  const speedSource = useMessageDataSource('speed')
 
  return (
    <React.Fragment>
      UI here
    </React.Fragment>
  )
}

LineChart, Console and PolledCSVLogger are some of the components which accept DataSources for time-series variable streams.

How it works

MessageDataSource is fed events by the QueryableMessageIDProvider which runs as part of the transport manager.

Additionally, the QueryableMessageIDProvider provides a method allowing the developer to manually process the incoming Message into an Event suitable for the timeseries system.

The default processor behaviour would like this (simplified for a example structure):

// @filename: template/src/transport-manager/index.tsx
import {
  ElectronIPCRemoteQueryExecutor,
  MultiPersistenceEngineMemory,
  QueryableMessageIDProvider,
} from '@electricui/core-timeseries'
import { Message } from '@electricui/core'
import { timing } from '@electricui/timing'
import { deviceManager } from './config'
 
const multiPersistenceEngine = new MultiPersistenceEngineMemory()
const remoteQueryExecutor = new ElectronIPCRemoteQueryExecutor(
  multiPersistenceEngine,
)
const queryableMessageIDProvider = new QueryableMessageIDProvider(
  deviceManager,
  multiPersistenceEngine,
)
 
queryableMessageIDProvider.setCustomMessageProcessor(
  'speed',
  (message, api) => {
    // Build the event at the current time
    api.emit(message.metadata.timestamp, {
      gps: message.payload.gps,
      wheels: message.payload.wheels,
    })
  },
)

Working with custom types

For projects using custom-typed data, the DataSource is able to infer the type correctly.

const speedDSTypeInfered = useMessageDataSource("speed")
              
const speedDSTypeInfered: MessageDataSource<SpeedSensors, "speed">

In practice, this allows for useful code-completion hints, and errors if an invalid accessor is written.

export type SpeedSensors = {
  gps: number
  wheels: number
}
 
const OverviewPage = () => {
  const speedDS = useMessageDataSource(MSGID.SPEED)
  return (
    <React.Fragment>
      <ChartContainer>
        <LineChart dataSource={speedDS} accessor={event => event.radar}/>
Property 'radar' does not exist on type 'SpeedSensors'.2339Property 'radar' does not exist on type 'SpeedSensors'.
        <RealTimeDomain window={10000} />
      </ChartContainer>
    </React.Fragment>
  )
}

If you're interested in how this works, read the deep-dive writeup on our blog!

If the type needs to be enforced manually, specify it globally with the TypedState file and use it as normal.

// /src/application/typedState.ts
export enum MSGID {
  SPEED = 'speed',
}
 
export type SpeedSensors = {
  gps: number
  wheels: number
}
 
declare global {
  interface ElectricUIDeveloperState {
    speed: SpeedSensors
  }
}

const speedDSTypeEnforced = useMessageDataSource(MSGID.SPEED)
              
const speedDSTypeEnforced: MessageDataSource<SpeedSensors, MSGID>

Retiming Events

By default, Events are timestamped by the transport manager.

In situations where a large amount of data is streaming from hardware, it is possible that dropped packets, a busy transport, or heavily loaded transport manager may not process data immediately as it arrives.

If this occurs, charts may show visual errors due to out-of-order packets or erroneous delays. The recommended approach in these circumstances is to provide hardware timing information in the packet.

// template/src/transport-manager/index.tsx
 
interface SpeedPayload {
  received: number // timestamp in ms since epoch, provided by hardware
  sensor: number // adc readings
}
queryableMessageIDProvider.setCustomMessageProcessor(
  'speed',
  (message: Message<SpeedPayload>, api) => {
    // the epoch needs to be subtracted to convert the global time provided by hardware to session time
    const epoch = timing.timeOrigin
 
    // Custom time from the message
    api.emit(message.payload!.received - epoch, message.payload!.sensor)
  },
)

It is also significantly more efficient to batch multiple events in the same packet.

By emitting multiple events for an inbound message, the downstream consumers can't tell a difference from a standard DataSource.

// template/src/transport-manager/index.tsx
 
interface PackedPayload {
  delta: number // the delta time between each subsequent event
  values: number[] // the adc values of each event
}
 
queryableMessageIDProvider.setCustomMessageProcessor(
  'speed',
  (message: Message<PackedPayload>, api) => {
    const start = message.metadata.timestamp
 
    if (!message.payload) {
      return
    }
 
    // Each packet has 512 sensor readings in an array
    // this API call emits each, aligned to the start time
    api.emitIntervalAlignStart(start, message.payload.delta, message.payload.values)
  },
)

Getting the current timestamp

In certain situations, knowing the timestamp for 'now' in the datasource or transformer can be useful, rather than the included packet timestamp.

The @electricui/timing package provides the timing.now() function, returning the current time in milliseconds since session start, and the timing.timeOrigin constant, storing the unix time in milliseconds since unix epoch of when the session began.

Timing is stored as floating point milliseconds since the start of the 'session' within the timeseries ecosystem.

Global time can be converted to session time using the timing.timeOrigin constant.

import { timing } from '@electricui/timing'
 
const timestamp = timing.now()
 
// timestamp of the start of session in milliseconds since unix epoch
const epoch = timing.timeOrigin 

Manipulating Data

Using a DataSource in isolation is fine when plotting raw signals streamed from hardware, but when 'client-side' signal processing or reformatting is required, custom transport-manager processors aren't suitable.

DataTransformers help you guide the data through a transformation pipeline with minimal loss of performance, whilst maintaing modular code.

import { leakyIntegrator } from '@electricui/dataflow'
import { useDataTransformer } from '@electricui/timeseries-react'
import { useMessageDataSource } from '@electricui/core-timeseries'
 
const OverviewPage = () => {
  const sensorDataSource = useMessageDataSource('adc')
 
  const filteredDS = useDataTransformer(() => {
    return leakyIntegrator(sensorDataSource, 4)
  })
 
  return (
    <React.Fragment>
      Chart of the averaged data
    </React.Fragment>
  )
}

The full guide with a range of worked examples is recommended reading:

Decimation

It is usually unnecessary to display more points than there are pixels on the screen.

To improve performance by default, all MessageDataSources have a "time stride" based decimator attached.

This time stride decimator buckets the data into as many sections as the chart is wide in pixels, and fetches the first Event within each bucket. This gives similar performance whether ten thousand or ten million events are stored in the persistence engine.

Decimation can be disabled via using a prepare DataFlow operator to set the query window limit to Infinity.

const undecimated = prepare(query => query.windowLimit(Infinity))

Persisting DataSources

Standalone, a DataSource only provides a stream of events. However, when a user leaves a page and eventually returns there's an implicit expectation that the graphed data will show the hardware's state while they were gone.

To achieve this, and the more complex functionality which powers decimation and the more complicated transformers, we persist the last 10k events with an in memory database.

Electric UI is configured with a balanced default setup, but we describe implementation details and how to customise the store in the Persistence guide.