Infrastructure#

Shepherd Nova comprises several distributed nodes and backend servers connected via Ethernet. Each node includes an observer and up to two devices under test, called targets. Typical targets are low-power IoT devices equipped with MCU, wireless transceivers, and peripherals such as sensors. The observers implement all services in hardware and software, emulating energy-harvesting conditions and collecting time-stamped observations. The backend servers

provide harvesting data for emulation to the observers,
store observations,
coordinate observer operations (e.g. starting and stopping experiments),
and interact with users who provide the target firmware and configuration of an experiment.

Please consult the paper for an in-depth system description. The paper also evaluates topics like faithfulness and repeatability - the cornerstones of the testbed.

**High-level architecture of Shepherd Nova:** Each testbed node consists of an observer and up to two targets. The observers are connected over Ethernet with multiple backend servers#

HW & SW architecture of an observer — Mapping of software modules to hardware components on a Shepherd Nova observer, showing the interaction and data flow between modules.#

Building a Chain of Trust#

For the design of the system we aimed for complete transparency and openness. But how can the user be sure that the testbed functioned as expected during an experiment? The following subsections are building on top of the topics already presented in the paper. We will describe our extensive warning-system which offers various checks and bounds. The goal was and is to supervise and self-diagnose every critical component during operation.

If something unplanned or bad happens, you will most likely hear about it.

Extensive logs are usually included in each result-file. Please note, that it’s not needed to comb through every entry to validate an experiment. These logs are mainly there for debugging in case something goes wrong. When using the provided shepherd-data utilities, the tool will usually report automatically if warnings and errors were emitted during the experiment.

With the additional goal of failing early, the testbed tries to verify states and configurations as soon as possible. The user can see that during the creation of an invalid experiment, i.e. when a target is requested twice by two separate target configurations. That concept was applied throughout the architecture.

Observer Logs#

For executing an experiment, three software units work together in unison on each observer:

the testbed service called shepherd-sheep which orchestrates all subsystems
a kernel module which coordinates hardware-access, data-flows and time-sync to the PRU-coprocessors
the firmware on the PRUs, running the virtual power source, gpio-tracer and accessing the analog frontend

Each of these systems can emit various warnings and errors which are recorded together with the verbose logs in the resulting hdf5-file. Operation of the kernel module is monitored by capturing the whole kernel log (dmesg). Same goes for the operation of the PRUs - which are monitored by and through the kernel module. Each PRU can send data and error-states via a dedicated message-system to the kernel module. Some messages / errors are only reported through the kernel log, but most errors are handed to the sheep-software and reported there.

Supervision#

Let’s have a look at the individual autonomous & supervised observer-tasks performed by shepherd-sheep:

patching the node-ID and converting the firmware from ELF to intel-hex
programming and validating the firmware written to each MCU on the target
starting, running and ending an experiment

It makes no sense to list every error-case, but to pick a few:

input files (energy-environments) are validated upon opening them
misbehaving synchronization between observers (PTP & PHC2SYS)
misbehaving sync between system and PRUs (which can be a result of a flaky outer sync)
a tracer that misses starting time or ends too early is detected
The supervisor for the buffers can detect backpressure, under- & overflows, and also harmed canaries between separate RAM-sections

Note

backpressure can occur when the gpio-tracer samples more data (capable of 1000+ kSps) than the filesystem can handle (~ 400 kSps) over a certain time. In that case buffer-sections need to be skipped to keep the system responsive. This will produce a timestamped error-log.

Statistics#

In addition to the discrete warnings each observer also records & timestamps utilization statistics:

PRU0 running the virtual source monitors it’s average and worst case execution time (WCET).
- The recorder will additionally warn about broken real-time conditions. This is important as it delays the following chip-select flank for triggering the ADC isochronously.
PRU1 running the GPIO-Tracer monitors the worst timedelta between reading GPIO-states.
- Shepherd guarantees 1 MSps, but often surpasses that.
- In comparison to tracing power, the sampling rate is not fixed.
- Just checking unchanged states currently (May 2025) results in 1.4 MHz minimal, 2.2 MHz mean and 2.9 MHz maximum rate.
- When states change and data has to be written, the rates will degrade to 1.1, 1.6 & 1.7 MHz respectively.
Linux allows recording system utilization, specifically from CPU, RAM, network-IO & filesystem-IO
Linux also allows recording synchronization statistics from NTP, PTP and PHC2SYS

Unittests & Quality Assurance#

Most software for this testbed is directly unit-tested during development. Tests that are hardware independent are also executed on GitHub via continuous integration workflows. Firmwares and other hard-to-test elements are at least compiled and / or run through a static analysis in these GitHub workflows. Merging pull-requests and drafting new releases is only possible when these workflows pass. This won’t make the system perfect, but at least catches a big chunk of possible flaws.

As an example for our testing - the software-stack of the observer offers a special loopback-test to validate data-consistency for the following route used by emulation-experiments:

    Input-File
      ↪ pySheep
        ↪ Input-RAM-Buffer
          ↪ Kernel Module
            ↪ OCMC-Buffer
              ↪ PRU0 (set to Loopback)
                ↪ Output-RAM-Buffer
                  ↪ pySheep
                    ↪ Output-File

By putting random data in and comparing the output to it, we can verify the whole stack at once.

Time Synchronization#

For running the testbed we are less interested in an absolute time consistency, but rather a relative sync between all observers. We aim for sub 1 µs accuracy and have chosen to use PTP on network level. The PTP-server is a Raspberry PI 4 that is first in its family to support hardware timestamping - an essential feature that corrects timestamps right before packets are sent onto the wire. The routing is handled by a single level 3 Cisco-switch (flat hierarchy). It uses a dedicated FPGA instead of a CPU to reduce latency and jitter. The switch itself supports no dedicated PTP-features like transparent clock.

Synchronization setup and results. — **Synchronization setup and results**: two nodes are synchronized to one PTP server over a standard Ethernet switch.#

Each observer is utilizing a BeagleBone - an SBC that is also capable of hardware timestamping. Via PTP & PHC2SYS the Linux system time is adjusted continuously. Each Shepherd observer records the statistics of both services in the result file to allow later diagnostics. In a last part of the chain, the system time is synchronized to the PRU-coprocessors. We have designed a dedicated self-tuning PI-controller that runs inside the kernel module and converges well within 20 seconds. The figure above shows the relative sync between two exemplary observers in the testbed. Signals were taken from the chip-select input of the ADC as it is precisely controlled by PRU0 for isochronous triggering. This measurement takes the whole synchronization-chain into account and exposes underlying flaws and misbehavior.

Virtual Power Source & Harvester#

The main routines for the virtual power source run on the PRU0-coprocessor. Besides being partly covered by unittests, these routines unfortunately act largely like a black box. Debugging the PRUs is hardly possible. To counteract the issue, we took four additional steps:

mathematical models, behaviors and parameters were explored, extended & validated during two master theses.
The virtual power source, including the harvester, was fully ported to python as a 1:1 functional-copy.
- A simulation can be run by adding a virtual target.
- Internal states and behaviors can be easily debugged similar to Simba. While Simba was not designed with real-time-capablities and hardware-in-the-loop in mind, both concepts have similar goals.
parts of the PRU-code can be tested by abstracting the analog frontend.
The C-code, running inside the PRU, can be compiled to a shared library with a c-types interface for python.
- Beside the python-port, this can act as an additional plugin-replacement for an emulation experiment.
- All three versions can be (and were) benchmarked against each other.

For more insight during the actual emulation the user can leverage the following config and data:

the power-good-signals (battery OK hysteresis) for the target is recorded together with the GPIO-Trace
it’s possible to record the voltage levels and output-current of the virtual energy storage instead of the direct target input
- helpful for configurations with static output voltage (i.e. buck-converter)

Calibration#

The analog frontend of each observer is calibrated with a Keithley 2604B SMU to allow precise measurements. Calibration data is stored on an EEPROM on the same PCB as the frontend and will be copied to the result-file during an experiment. Additionally, the cal-data is stored online to avoid loss and enable quick lookup. During recording the raw values from the ADCs are not altered, but stored as is in the hdf5-file. This would allow for later re-calibration and easily correcting the result-files, in case an issue is found with an individual calibration. Calibrations made two years apart showed excellent long time stability of the frontend. More details can be found in the paper.

Benchmarks#

To give a feeling for what’s possible, we ran several 1-minute performance-analysis on the observers. The results will be updated after major changes.

Due to using fully random data these numbers should be a worst-case estimation. Every result-file contains cpu-utilization and other stats that can be extracted with:

shepherd-data extract-meta --debug ./file

A mean cpu-utilization below 90 % is considered “safe” for the testbed. The current software is heavily optimized and should stay in line when using the system within specs. As all recorded data on the observers is buffered in RAM it is possible to exceed thresholds within burst or in short-term experiments. For more headroom it is possible to either disable Power-Tracing (notrace-entry) or use it with a custom lower sample-rate.

Recordings for GPIO and UART were benchmarked with a special high-throughput uart-firmware on the targets. The Readme explains the actual rates and the methodology.

More critical is the load on the network. The overload is caused by excessive write-rates and users see failed experiments as a symptom or result. When the testbed reports high PTP-Sync offsets, it is most likely caused by a network congestion.

Benchmark-data is taken from shepherd-main. The document contains raw data from before, in between & after.

Benchmark 2026-10-23 - Second Perf-Update#

TLDR: CPU-Utilization is 25-40 %pt below baseline; GPIO-Tracing with fast data-rates is now possible;

Type	CPU_min [%]	CPU_max [%]	CPU_mean [%]	File [MiB/s]
EMU IV ShpInput ivtrace	30	73	45.0	0.475
EMU IV ShpInput notrace	11	36	18.3	0.001
EMU IV hrvformat ivtrace	23	80	42.4	0.475
EMU IV hrvformat notrace	12	36	17.0	0.001
EMU GPIO datarate=57600	22	87	38.0	0.309
EMU GPIO datarate=115200	29	87	41.8	0.392
EMU GPIO datarate=230400	30	100	50.4	0.546
EMU GPIO datarate=460800	33	100	63.0	0.787
EMU GPIO datarate=921600	18	100	81.8	1.146
EMU GPIO datarate=1000000	10	100	85.2	1.238
EMU Uart datarate=57600	41	71	57.0	0.231
EMU Uart datarate=115200	31	80	59.5	0.240
EMU Uart datarate=230400	42	87	62.8	0.252
EMU Uart datarate=460800	60	88	69.0	0.270
EMU Power @ samplerate=10	20	41	30.5	0.001
EMU Power @ samplerate=100	21	40	31.2	0.002
EMU Power @ samplerate=1000	19	45	30.3	0.005
EMU Power @ samplerate=10000	22	44	31.9	0.041
EMU Power @ samplerate=100000	26	92	43.3	0.370
EMU IV @ samplerate=10	18	34	27.9	0.001
EMU IV @ samplerate=100	17	46	28.7	0.002
EMU IV @ samplerate=1000	14	42	28.9	0.006
EMU IV @ samplerate=10000	11	45	31.1	0.050
EMU IV @ samplerate=100000	31	72	43.9	0.476

Benchmark 2026-10-15 - First Perf-Update#

TLDR: Filesize was roughly halved; CPU-Utilization decreases by ~ 10 %pt compared to baseline;

Type	CPU_min [%]	CPU_max [%]	CPU_mean [%]	File [MiB/s]	Note
EMU IV ShpInput ivtrace	25	83	61.0	0.475
EMU IV ShpInput notrace	15	60	34.7	0.002
EMU IV hrvformat ivtrace	31	86	61.1	0.476
EMU IV hrvformat notrace	23	55	38.8	0.002
EMU GPIO datarate=57600	50	95	65.2	0.310
EMU GPIO datarate=115200	55	100	70.0	0.392
EMU GPIO datarate=230400	65	100	79.0	0.547
EMU GPIO datarate=460800	77	100	92.7	0.787	high load
EMU GPIO datarate=921600	78	100	98.9	1.146	high load
EMU GPIO datarate=1000000	77	100	99.1	1.238	high load
EMU Power @ samplerate=10	42	73	59.1	0.002
EMU Power @ samplerate=100	33	79	60.1	0.002
EMU Power @ samplerate=1000	49	72	63.0	0.006
EMU Power @ samplerate=100000	48	86	66.5	0.357
EMU IV @ samplerate=10	35	73	51.4	0.002
EMU IV @ samplerate=100	37	61	51.2	0.002
EMU IV @ samplerate=1000	36	61	51.3	0.006
EMU IV @ samplerate=100000	44	96	65.9	0.477

Benchmark 2026-10-07 - Baseline#

Type	CPU_min [%]	CPU_max [%]	CPU_mean [%]	File [MiB/s]	Note
EMU IV ShpInput ivtrace	54	100	72.5	1.087	had errors, incomplete
EMU IV ShpInput notrace	31	100	44.5	0.002
EMU Power & samplerate=10	47	100	63.8	0.003
EMU Power & samplerate=100	52	93	64.3	0.003
EMU Power & samplerate=1000	57	100	69.5	0.010
EMU Power & samplerate=100000	68	100	82.8	0.501
EMU IV & samplerate=10	42	88	56.7	0.003
EMU IV & samplerate=100	44	100	58.6	0.003
EMU IV & samplerate=1000	44	100	58.6	0.010
EMU IV & samplerate=100000	66	100	79.1	0.687

Getting started

Energy Environments