dominic/st
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial release of Streaming Event Trace

Browse Source
This commit is contained in:
Dominic Höglinger 2025-04-28 22:09:19 +02:00
commit 5c3473ff1f
5 changed files with 1821 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
LICENSE.md Normal file
View File

@ -0,0 +1,21 @@
MIT License
Copyright (c) 2025 Dominic Hoeglinger
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

100
README.md Normal file
View File

@ -0,0 +1,100 @@
```
▓▓▓▓▓ ░░░ ███████ ███████ ████████
▓ ▓ ▒ ░ ██ ██ ██ Streaming Event Trace
▓ ▓ ▓ ░░ ███████ █████ ██ A tiny signal tracing library
▓ ▓ ▒ ░ ██ ██ ██
▓▓▓▓▓ ░░░ ███████ ███████ ██ Version 1.0.1 Alpha
```
# Introduction
SET is a small tracing library written in C99.
It is highly optimized for minimal data usage and aims to be easy to use
while also being flexible enough to be integrated in hard real time systems.
# How to use
SET requires the integrator to implement a few functions such as the timestamp and output functions in order to work.
Apart from that, it is up to the specific use case how to stream data.
In its simplest form, traces are processed with no real time demands on a dedicated interface.
For more advanced usage, this library allows packets to be written, as the time budget allows,
into any given buffer to send its contents via DMA to for instance a serial output.
Simple example on Arduino:
````c
#include "set.h"
// Basic implementation of output and timestamp retrieval
void set_out(const char* const str, size_t len) { Serial.write(str, len); }
uint32_t set_timestamp(void) { return micros(); }
// No concurrency protection needed
void set_crit_on(uint32_t *h) {}
void set_crit_off(const uint32_t h) {}
static const size_t s_tracesize = 256;
static set_trace_t s_tracebuf[256];
void setup()
{
set_init(s_tracebuf, s_tracesize);
set_enable(set_drop); // drop traces on full buffer
set_u8trace("Serial.Ready", 0, false);
Serial.begin(921600);
while (!Serial);
set_u8trace("Serial.Ready", 1, false);
}
void loop()
{
unsigned sawtooth = 0;
if (sawtooth < 0xFFFF) sawtooth++;
else sawtooth = 0;
set_u32trace("Main.Sawtooth", sawtooth, false);
if (sawtooth == 4711) Serial.write("I can still be used with regular text.");
set_output(0, true); //send unlimited traces compressed
set_diagtrace(); // add diagnostic traces
}
````
# Basic operating principle
Traces are identified by a signal name, which may be hierarchical such as `ExampleModule.Function.Value`.
These must be constant C strings as their pointers are stored in the trace buffer.
Once a trace is rendered, the signal name is hashed using the BSD sum function.
This allows a client to identify a trace packet according to the source tree scanned.
For sending the traces to a client, they are rendered into a packet.
A packet consist of a preamble, a flag byte, frame data and an epilouge.
The preamble and epilouge are chosen to be the ANSI escape sequences for storing the current position, and restoring it.
The frame data may be compressed using fastlz, which is indicated by the Z flag in the packet.
Regardless of compression, it is COBS encoded with delimiter 0033 to not interfere with the preamble or epilouge.
Frame data consists of multiple frames, COBS encoded with delimiter 0 and delimited with the same value.
Each frame contains the signal hash, its type and depending on it a value ranging from 1-4 bytes and indexing information.
Timestamps are sent in increments to conserve bandwidth.
All trace functions feature a parameter which enables timestamp capture, if it is omitted this means no time delta is sent,
making the trace appear to coincide with the last trace which captured a timestamp.
# The recorder utility
The repository contains an example client application which captures packages and outputs the traces in a Value Change Dump.
This script uses the PyVCD package which is the only dependency.
Apart from that, this utility has every function integrated it'll need to decode a trace stream.
It expects data via stdin, and displays any non-tracing traffic.
Example usage tracing remotely via SSH:
````bash
# Open /dev/ttyUSB0 via SSH and record to "trace.vcd" with timescale of 1 microsecond with diagnostics enabled
ssh user@host -C "picocom -b 921600 /dev/ttyUSB0" | ./set_record.py -d trace.vcd -s ./Sources -t "1 us" --diagnostics
````
The resulting VCD can be inspected with dedicated programs such as GTKwave.

872
set.c Normal file
View File

@ -0,0 +1,872 @@
/*
* PET: Pocket Event Trace - A tiny tracing library
* Copyright (C) 2025 Dominic Hoeglinger
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#include "pet.h"
#define SET_NIBBLE_TO_HEX(nibble) ((nibble) < 10 ? (nibble) + '0' : (nibble) - 10 + 'A')
#define SET_NIBBLE(value, nibble) ((value >> nibble) & 0xF)
#define SET_BYTE(value, offset) (char)(((value >> (8 * offset)) & 0xFF))
#define SET_ENCODE(value, nibble) (char)(SET_NIBBLE_TO_HEX(SET_NIBBLE(value, nibble)))
#define SET_NELEMS(a) (sizeof(a) / sizeof(a[0]))
#define SET_MACROPACK_FRAME_SIZE (2 * 8)
#define SET_MACROPACK_SIZE(n) (SET_MACROPACK_FRAME_SIZE * n + (SET_MACROPACK_FRAME_SIZE * n) / 254 + 1 + n - 1)
#define SET_MAX_RENDER_SIZE(n) (SET_MACROPACK_SIZE(n) + (SET_MACROPACK_SIZE(n) / 254) + 1)
#define FASTLZ1_MAX_COPY 32
#define FASTLZ1_MAX_LEN 264 /* 256 + 8 */
#define FASTLZ1_MAX_L1_DISTANCE 8192
#define FASTLZ1_MAX_L2_DISTANCE 8191
#define FASTLZ1_MAX_FARDISTANCE (65535 + FASTLZ1_MAX_L2_DISTANCE - 1)
#define FASTLZ1_HASH_LOG 13
#define FASTLZ1_HASH_SIZE (1 << FASTLZ1_HASH_LOG)
#define FASTLZ1_HASH_MASK (FASTLZ1_HASH_SIZE - 1)
typedef enum
{
set_d1 = 0x11,
set_d2 = 0x12,
set_d4 = 0x14,
set_v1 = 0x21,
set_v2 = 0x22,
set_v4 = 0x24,
set_a1 = 0x31,
set_a2 = 0x32,
set_a4 = 0x34,
set_s4 = 0x44,
set_f4 = 0x54,
set_ev = 0x60,
} set_type_t;
typedef enum
{
set_packet_z = (1 << 0),
} set_packetflags_t;
typedef struct
{
const char* m_tag;
uint16_t m_hash;
} set_hashcache_t;
typedef union
{
uint32_t u32value;
#if (!SET_FLOAT_DISABLE)
float f32value;
#endif
} set_convert_t;
static bool s_enabled = false;
static uint32_t s_last_time = 0;
set_overflow_policy_t s_policy = set_overwrite;
static set_trace_t* s_tracebuffer;
static size_t s_tracebuffer_size = 0;
static size_t s_tracebuffer_head = 0;
static size_t s_tracebuffer_tail = 0;
static size_t s_tracebuffer_items = 0;
static bool s_tracebuffer_full = false;
static bool s_locked = false;
static bool s_tty_enabled = false;
static size_t s_tty_rubout_len = 0;
static uint32_t s_diag_avg_compression_total = 0;
static uint32_t s_diag_avg_compression_n = 0;
static uint32_t s_diag_items_sent = 0;
static uint32_t s_diag_avg_ctime_total = 0;
static uint32_t s_diag_avg_ctime_n = 0;
static uint32_t s_diag_avg_rtime_total = 0;
static uint32_t s_diag_avg_rtime_n = 0;
#if (SET_HASHCACHE_LINES > 0)
static set_hashcache_t s_hashcache[SET_HASHCACHE_LINES] = {0};
static size_t s_hashcache_index = 0;
#endif
size_t render_macropacket_payload(char buffer[], const size_t buffer_size, size_t n);
static size_t frame_preamble(char buffer[], const size_t buffer_size);
static size_t frame_epilouge(char buffer[], const size_t buffer_size);
static size_t pack_frame(char packet_bytes[], const set_trace_t* const p_packet);
static size_t ttyize_payload(char packet_buffer[], const size_t packet_size, const size_t buffer_size);
static uint16_t bsd_hash(const char* const str, size_t n);
static bool populate_time_delta_packet(const uint32_t timestamp, set_trace_t* const p_trace);
static void tracebuffer_add(const uint32_t timestamp, const set_type_t type, const uint32_t value, const uint8_t sub, char const *tag);
size_t cobs_encode(uint8_t* dst, size_t dst_buf_len, const uint8_t* ptr, int len, uint8_t delim);
static int fastlz1_compress(const void* input, int length, void* output);
void set_init(set_trace_t tracebuffer[], size_t ntraces)
{
s_tracebuffer = tracebuffer;
s_tracebuffer_size = ntraces;
}
size_t set_get_buffer_items(void)
{
return s_tracebuffer_items;
}
void set_tty_mode(const bool enable)
{
s_tty_enabled = enable;
}
size_t set_tty_rubout(char buffer[], const size_t buffer_size)
{
const size_t rubout_packet_size = 3 + s_tty_rubout_len + 3;
size_t buffer_pos = 0;
if (buffer_size < rubout_packet_size)
{
return 0;
}
buffer_pos += frame_preamble(buffer, buffer_size);
for (size_t i = 0; i < s_tty_rubout_len; ++i)
{
buffer[buffer_pos + i] = ' ';
}
buffer_pos += s_tty_rubout_len;
buffer_pos += frame_epilouge(&buffer[buffer_pos], buffer_size - buffer_pos);
s_tty_rubout_len = 0;
return buffer_pos;
}
size_t set_output(size_t n, const bool compress)
{
const size_t render_number_items = 4;
const size_t render_max_size = SET_MAX_RENDER_SIZE(render_number_items);
char packet_bytes[SET_MAX_RENDER_SIZE(4)] = {0};
size_t packet_size = 0;
size_t i = 0;
n = (n == 0) ? s_tracebuffer_size : n;
while ((s_tracebuffer_full || (s_tracebuffer_tail != s_tracebuffer_head)) && (i < n))
{
packet_size = set_render(packet_bytes, render_max_size, render_number_items, compress);
i += render_number_items;
set_out(packet_bytes, packet_size);
}
packet_size = set_tty_rubout(packet_bytes, render_max_size);
set_out(packet_bytes, packet_size);
return i;
}
size_t render_macropacket_payload(char buffer[], const size_t buffer_size, size_t n)
{
const size_t max_render_size = (8 + 1 + 1) * 2 - 1;
char payload_buffer[8] = {0};
char delimimter = 0x00;
set_trace_t timestamp_pack = {0};
size_t payload_size = 0;
size_t i = 0;
size_t buffer_pos = 0;
size_t encode_size = 0;
n = (n == 0) ? s_tracebuffer_size : n;
while ((s_tracebuffer_full || (s_tracebuffer_tail != s_tracebuffer_head)) && (i < n) && ((buffer_pos + max_render_size) < buffer_size))
{
payload_size = pack_frame(payload_buffer, &s_tracebuffer[s_tracebuffer_tail]);
encode_size = cobs_encode(&buffer[buffer_pos], buffer_size - buffer_pos, payload_buffer, payload_size, delimimter);
buffer_pos += encode_size;
buffer[buffer_pos] = delimimter;
buffer_pos++;
if (populate_time_delta_packet(s_tracebuffer[s_tracebuffer_tail].m_timestamp, &timestamp_pack))
{
payload_size = pack_frame(payload_buffer, &timestamp_pack);
encode_size = cobs_encode(&buffer[buffer_pos], buffer_size - buffer_pos, payload_buffer, payload_size, delimimter);
buffer_pos += encode_size;
buffer[buffer_pos] = delimimter;
buffer_pos++;
}
s_tracebuffer_tail = (s_tracebuffer_tail + 1) % s_tracebuffer_size;
s_tracebuffer_full = false;
s_tracebuffer_items = (s_tracebuffer_items > 0) ? (s_tracebuffer_items - 1) : 0;
i++;
s_diag_items_sent++;
}
// remove last delimiter
if(buffer_pos) buffer_pos--;
return buffer_pos;
}
size_t set_render(char buffer[], const size_t buffer_size, const size_t n, const bool compress)
{
const char delimimter = '\033';
size_t n_elems = (n == 0) ? s_tracebuffer_items : n;
n_elems = (s_tty_enabled && (n > 4)) ? 4 : n_elems;
size_t render_max_size = SET_MAX_RENDER_SIZE(n_elems);
uint32_t ticks_render_start = set_timestamp();
char *final_payload_buffer = NULL;
size_t final_payload_size = 0;
while ((buffer_size < (1 + (render_max_size * 2))) && (n_elems > 0))
{
n_elems--;
render_max_size = SET_MAX_RENDER_SIZE(n_elems);
}
if (n_elems == 0) return 0;
if (compress)
{
size_t macropack_payload_size = render_macropacket_payload(buffer, render_max_size, n_elems);
if (macropack_payload_size == 0) return 0;
size_t compressed_size = macropack_payload_size + (macropack_payload_size*15)/100;
char *compressed_buffer = &buffer[buffer_size - compressed_size];
uint32_t ticks_comp_start = set_timestamp();
compressed_size = fastlz1_compress(buffer, macropack_payload_size, compressed_buffer);
final_payload_buffer = compressed_buffer;
final_payload_size = compressed_size;
s_diag_avg_ctime_total += set_timestamp() - ticks_comp_start;
s_diag_avg_ctime_n++;
if (macropack_payload_size > 0)
{
s_diag_avg_compression_total += 100 - ((100 * compressed_size) / macropack_payload_size);
s_diag_avg_compression_n++;
}
}
else
{
char *macropack_buffer = &buffer[buffer_size - render_max_size];
size_t macropack_payload_size = render_macropacket_payload(macropack_buffer, render_max_size, n_elems);
if (macropack_payload_size == 0) return 0;
final_payload_buffer = macropack_buffer;
final_payload_size = macropack_payload_size;
}
size_t out_pos = frame_preamble(buffer, buffer_size);
uint8_t flags = 0;
if (compress) flags |= (uint8_t)set_packet_z;
buffer[out_pos] = flags;
out_pos++;
size_t payload_size = cobs_encode(&buffer[out_pos], buffer_size - out_pos, final_payload_buffer, final_payload_size, delimimter);
if (s_tty_enabled == true)
{
payload_size = ttyize_payload(&buffer[out_pos], payload_size, buffer_size - out_pos);
if (payload_size > s_tty_rubout_len)
{
s_tty_rubout_len = payload_size;
}
}
out_pos += payload_size;
out_pos += frame_epilouge(&buffer[out_pos], buffer_size - out_pos);
s_diag_avg_rtime_total += set_timestamp() - ticks_render_start;
s_diag_avg_rtime_n++;
return out_pos;
}
void set_enable(const set_overflow_policy_t policy)
{
s_policy = policy;
s_enabled = true;
}
void set_disable(void)
{
s_enabled = false;
}
void set_clear(void)
{
s_tracebuffer_tail = s_tracebuffer_head;
s_tracebuffer_full = false;
}
void set_diagtrace(void)
{
const uint32_t buffer_health = ((s_tracebuffer_size - s_tracebuffer_items) * 0xFF)/s_tracebuffer_size;
set_u32trace("PET.BufferItems", s_tracebuffer_items, false);
set_u8trace("PET.BufferHealth", (uint8_t)buffer_health, true);
if (s_diag_avg_compression_n > 0)
{
const uint32_t average = s_diag_avg_compression_total / s_diag_avg_compression_n;
set_u8trace("PET.CompressionLevel", (uint8_t)average, true);
s_diag_avg_compression_total = 0;
s_diag_avg_compression_n = 0;
}
if (s_diag_items_sent > 0)
{
set_u32trace("PET.ItemsSent", s_diag_items_sent, true);
s_diag_items_sent = 0;
}
if (s_diag_avg_ctime_n > 0)
{
const uint32_t average = s_diag_avg_ctime_total / s_diag_avg_ctime_n;
set_u8trace("PET.CompressionTime", (uint8_t)average, true);
s_diag_avg_ctime_total = 0;
s_diag_avg_ctime_n = 0;
}
if (s_diag_avg_rtime_n > 0)
{
const uint32_t average = s_diag_avg_rtime_total / s_diag_avg_rtime_n;
set_u8trace("PET.RenderTime", (uint8_t)average, true);
s_diag_avg_rtime_total = 0;
s_diag_avg_rtime_n = 0;
}
}
void set_evtrace(const char* const tag, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_ev, 1UL, 0U, tag);
}
}
void set_u8trace(const char* const tag, const uint8_t value, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_v1, (uint32_t)value, 0U, tag);
}
}
void set_u16trace(const char* const tag, const uint16_t value, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_v2, (uint32_t)value, 0U, tag);
}
}
void set_u32trace(const char* const tag, const uint32_t value, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_v4, (uint32_t)value, 0U, tag);
}
}
void set_s8trace(const char* const tag, const int8_t value, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_v1, (uint32_t)value, 0U, tag);
}
}
void set_s16trace(const char* const tag, const int16_t value, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_v2, (uint32_t)value, 0U, tag);
}
}
void set_s4trace(const char* const tag, const int32_t value, const bool skip_time)
{
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_v4, (uint32_t)value, 0U, tag);
}
}
void set_au8trace(const char* const tag, const uint8_t value[const], const uint8_t size, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
for (size_t i = 0; i < size; ++i)
{
tracebuffer_add(ts, set_a1, (uint32_t)value[i], (uint8_t)i, tag);
}
}
}
void set_au16trace(const char* const tag, const uint16_t value[const], const uint8_t size, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
for (size_t i = 0; i < size; ++i)
{
tracebuffer_add(ts, set_a2, (uint32_t)value[i], (uint8_t)i, tag);
}
}
}
void set_au32trace(const char* const tag, const uint32_t value[const], const uint8_t size, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
for (size_t i = 0; i < size; ++i)
{
uint16_t sub = (i) | (size << 8);
tracebuffer_add(ts, set_a4, (uint32_t)value[i], (uint8_t)i, tag);
}
}
}
void set_as8trace(const char* const tag, const int8_t value[const], const uint8_t size, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
for (size_t i = 0; i < size; ++i)
{
tracebuffer_add(ts, set_a1, (uint32_t)value[i], (uint8_t)i, tag);
}
}
}
void set_as16trace(const char* const tag, const int16_t value[const], const uint8_t size, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
for (size_t i = 0; i < size; ++i)
{
tracebuffer_add(ts, set_a2, (uint32_t)value[i], (uint8_t)i, tag);
}
}
}
void set_as32trace(const char* const tag, const int32_t value[const], const uint8_t size, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
for (size_t i = 0; i < size; ++i)
{
uint16_t sub = (i) | (size << 8);
tracebuffer_add(ts, set_a4, (uint32_t)value[i], (uint8_t)i, tag);
}
}
}
void set_strtrace(const char* const tag, const char* const str, const bool skip_time)
{
if (s_enabled == true)
{
const size_t ts = skip_time ? 0UL : set_timestamp();
uint32_t packword = 0;
uint8_t packindex = 0;
for (size_t i = 0; str[i] != '\0'; ++i)
{
packindex = (i % 4);
packword |= (str[i] << (8 * packindex));
if (packindex == 3)
{
tracebuffer_add(ts, set_s4, packword, (uint8_t)str[i + 1] == '\0', tag);
packword = 0;
}
}
if (packindex != 3)
{
tracebuffer_add(ts, set_s4, packword, (uint8_t)true, tag);
}
}
}
#if (!SET_FLOAT_DISABLE)
void set_f32trace(const char* const tag, const float value, const bool skip_time)
{
set_convert_t converter = {.f32value = value };
if (s_enabled == true)
{
tracebuffer_add(skip_time ? 0UL : set_timestamp(), set_f4, converter.u32value, 0U, tag);
}
}
#endif
static size_t frame_preamble(char buffer[], const size_t buffer_size)
{
size_t pack_pos = 0;
if (buffer_size >= 3)
{
buffer[pack_pos++] = '\033';
buffer[pack_pos++] = '[';
buffer[pack_pos++] = 's';
}
return pack_pos;
}
static size_t frame_epilouge(char buffer[], const size_t buffer_size)
{
size_t pack_pos = 0;
if (buffer_size >= 3)
{
buffer[pack_pos++] = '\033';
buffer[pack_pos++] = '[';
buffer[pack_pos++] = 'u';
}
return pack_pos;
}
static size_t pack_frame(char packet_bytes[], const set_trace_t* const p_packet)
{
size_t packet_size = 0;
packet_bytes[packet_size++] = (char)p_packet->m_type;
switch ((set_type_t)p_packet->m_type)
{
case set_v1:
case set_a1:
case set_d1:
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 0);
break;
case set_v2:
case set_a2:
case set_d2:
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 0);
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 1);
break;
case set_v4:
case set_a4:
case set_s4:
case set_f4:
case set_d4:
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 0);
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 1);
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 2);
packet_bytes[packet_size++] = SET_BYTE(p_packet->m_value, 3);
break;
}
switch ((set_type_t)p_packet->m_type)
{
case set_a1:
case set_a2:
case set_a4:
case set_s4:
packet_bytes[packet_size++] = (char)p_packet->m_sub;
break;
}
switch ((set_type_t)p_packet->m_type)
{
case set_d1:
case set_d2:
case set_d4:
break;
default:
const uint16_t hash_tag = bsd_hash(p_packet->m_tag, SET_MAX_TAG_LEN);
packet_bytes[packet_size++] = SET_BYTE(hash_tag, 0);
packet_bytes[packet_size++] = SET_BYTE(hash_tag, 1);
break;
}
return packet_size;
}
static size_t ttyize_payload(char packet_buffer[], const size_t packet_size, const size_t buffer_size)
{
const size_t tty_packet_size = (packet_size * 2);
size_t backpos = 0;
char value = 0;
if (tty_packet_size > buffer_size) return 0;
for (size_t i = 0; i < packet_size; ++i)
{
value = packet_buffer[(packet_size - i - 1)];
backpos = (packet_size - i - 1) * 2;
packet_buffer[backpos] = SET_ENCODE(value, 0);
packet_buffer[backpos + 1] = SET_ENCODE(value, 4);
}
return tty_packet_size;
}
static uint16_t bsd_hash(const char* const str, size_t n)
{
uint32_t checksum = 0;
if (str == NULL) return 0U;
#if (SET_HASHCACHE_LINES > 0)
for (size_t i = 0; i < SET_HASHCACHE_LINES; ++i)
{
size_t hash_index = (i + s_hashcache_index) % SET_HASHCACHE_LINES;
if (s_hashcache[hash_index].m_tag == str)
{
return s_hashcache[hash_index].m_hash;
}
}
#endif
for (size_t i = 0; (i < n) && (str[i] != '\0'); ++i)
{
checksum = (checksum >> 1) + ((checksum & 1) << 15);
checksum += str[i];
checksum &= 0xFFFF;
}
#if (SET_HASHCACHE_LINES > 0)
s_hashcache_index = (s_hashcache_index + 1) % SET_HASHCACHE_LINES;
s_hashcache[s_hashcache_index].m_hash = checksum;
s_hashcache[s_hashcache_index].m_tag = str;
#endif
return checksum;
}
static bool populate_time_delta_packet(const uint32_t timestamp, set_trace_t* const p_trace)
{
const uint32_t dt = timestamp - s_last_time;
uint16_t type = 0;
if (timestamp == 0UL) return false;
if (dt != 0)
{
if (dt > 0xFFFF)
{
type = set_d4;
}
else if (dt > 0xFF)
{
type = set_d2;
}
else
{
type = set_d1;
}
p_trace->m_type = type;
p_trace->m_value = dt;
p_trace->m_sub = 0U;
p_trace->m_tag = NULL;
p_trace->m_timestamp = 0UL;
s_last_time = timestamp;
return true;
}
return false;
}
static void tracebuffer_add(const uint32_t timestamp, const set_type_t type, const uint32_t value, const uint8_t sub, char const *tag)
{
const size_t insertion_pos = s_tracebuffer_head;
uint32_t critical_h = 0UL;
if (s_tracebuffer_full == false || s_policy == set_overwrite)
{
set_crit_on(&critical_h);
s_tracebuffer_head = (s_tracebuffer_head + 1) % s_tracebuffer_size;
s_tracebuffer_full = (s_tracebuffer_head == s_tracebuffer_tail);
s_tracebuffer_items++;
set_crit_off(critical_h);
s_tracebuffer[insertion_pos].m_type = (uint8_t)type;
s_tracebuffer[insertion_pos].m_value = value;
s_tracebuffer[insertion_pos].m_sub = sub;
s_tracebuffer[insertion_pos].m_tag = tag;
s_tracebuffer[insertion_pos].m_timestamp = timestamp;
}
}
size_t cobs_encode(uint8_t* dst, size_t dst_buf_len, const uint8_t* ptr, int len, uint8_t delim)
{
size_t encode_pos = 1;
size_t ins = 0;
uint8_t code = 1;
bool add_last_code = true;
size_t dst_size = 0;
for (size_t i = 0; i < len; ++i)
{
uint8_t byte = ptr[i];
if (byte != 0)
{
dst[encode_pos] = byte ^ delim;
encode_pos++;
code = code + 1;
}
add_last_code = true;
if ((byte == 0) || (code == 0xFF))
{
if (code == 0xFF) add_last_code = false;
dst[ins] = code ^ delim;
code = 1;
ins = encode_pos;
dst[encode_pos] = 0xFF ^ delim;
encode_pos++;
}
}
if (add_last_code)
{
dst[ins] = code ^ delim;
dst[encode_pos] = delim;
encode_pos++;
}
else
{
dst[ins] = delim;
}
encode_pos--;
return encode_pos;
}
static uint32_t flz_readu32(const void* ptr) {
const uint8_t* p = (const uint8_t*)ptr;
return (p[3] << 24) | (p[2] << 16) | (p[1] << 8) | p[0];
}
static uint16_t flz_hash(uint32_t v) {
uint32_t h = (v * 2654435769LL) >> (32 - FASTLZ1_HASH_LOG);
return h & FASTLZ1_HASH_MASK;
}
static void set_memcpy(uint8_t* dest, const uint8_t* src, uint32_t count)
{
for(size_t i = 0; i < count; ++i)
{
dest[i] = src[i];
}
}
static uint32_t flz_cmp(const uint8_t* p, const uint8_t* q, const uint8_t* r) {
const uint8_t* start = p;
while (q < r)
if (*p++ != *q++) break;
return p - start;
}
static uint8_t* flz_literals(uint32_t runs, const uint8_t* src, uint8_t* dest) {
while (runs >= FASTLZ1_MAX_COPY) {
*dest++ = FASTLZ1_MAX_COPY - 1;
set_memcpy(dest, src, FASTLZ1_MAX_COPY);
src += FASTLZ1_MAX_COPY;
dest += FASTLZ1_MAX_COPY;
runs -= FASTLZ1_MAX_COPY;
}
if (runs > 0) {
*dest++ = runs - 1;
set_memcpy(dest, src, runs);
dest += runs;
}
return dest;
}
static uint8_t* flz1_match(uint32_t len, uint32_t distance, uint8_t* op) {
--distance;
if (len > FASTLZ1_MAX_LEN - 2)
while (len > FASTLZ1_MAX_LEN - 2) {
*op++ = (7 << 5) + (distance >> 8);
*op++ = FASTLZ1_MAX_LEN - 2 - 7 - 2;
*op++ = (distance & 255);
len -= FASTLZ1_MAX_LEN - 2;
}
if (len < 7) {
*op++ = (len << 5) + (distance >> 8);
*op++ = (distance & 255);
} else {
*op++ = (7 << 5) + (distance >> 8);
*op++ = len - 7;
*op++ = (distance & 255);
}
return op;
}
static int fastlz1_compress(const void* input, int length, void* output) {
const uint8_t* ip = (const uint8_t*)input;
const uint8_t* ip_start = ip;
const uint8_t* ip_bound = ip + length - 4; /* because readU32 */
const uint8_t* ip_limit = ip + length - 12 - 1;
uint8_t* op = (uint8_t*)output;
uint32_t htab[FASTLZ1_HASH_SIZE];
uint32_t seq, hash;
/* initializes hash table */
for (hash = 0; hash < FASTLZ1_HASH_SIZE; ++hash) htab[hash] = 0;
/* we start with literal copy */
const uint8_t* anchor = ip;
ip += 2;
/* main loop */
while (ip < ip_limit) {
const uint8_t* ref;
uint32_t distance, cmp;
/* find potential match */
do {
seq = flz_readu32(ip) & 0xffffff;
hash = flz_hash(seq);
ref = ip_start + htab[hash];
htab[hash] = ip - ip_start;
distance = ip - ref;
cmp = (distance < FASTLZ1_MAX_L1_DISTANCE) ? flz_readu32(ref) & 0xffffff : 0x1000000;
if (ip >= ip_limit) break;
++ip;
} while (seq != cmp);
if (ip >= ip_limit) break;
--ip;
if (ip > anchor) {
op = flz_literals(ip - anchor, anchor, op);
}
uint32_t len = flz_cmp(ref + 3, ip + 3, ip_bound);
op = flz1_match(len, distance, op);
/* update the hash at match boundary */
ip += len;
seq = flz_readu32(ip);
hash = flz_hash(seq & 0xffffff);
htab[hash] = ip++ - ip_start;
seq >>= 8;
hash = flz_hash(seq);
htab[hash] = ip++ - ip_start;
anchor = ip;
}
uint32_t copy = (uint8_t*)input + length - anchor;
op = flz_literals(copy, anchor, op);
return op - (uint8_t*)output;
}

285
set.h Normal file
View File

@ -0,0 +1,285 @@
/*
* PET: Pocket Event Trace - A tiny tracing library
* Copyright (C) 2025 Dominic Hoeglinger
*
* Permission is hereby granted, free of charge, to any person obtaining a copy
* of this software and associated documentation files (the "Software"), to deal
* in the Software without restriction, including without limitation the rights
* to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
* copies of the Software, and to permit persons to whom the Software is
* furnished to do so, subject to the following conditions:
*
* The above copyright notice and this permission notice shall be included in
* all copies or substantial portions of the Software.
*
* THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
* IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
* FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
* AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
* LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
* OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
* SOFTWARE.
*/
#ifndef SET_H_INCLUDED_
#define SET_H_INCLUDED_
#include <stddef.h>
#include <stdbool.h>
#include <stdint.h>
/** Maxmum signal name / tag length considered for the identifying hash */
#define SET_MAX_TAG_LEN (126)
/** Number of cached hashes */
#define SET_HASHCACHE_LINES (64)
/** Switch to enable or disable floating point support */
#define SET_FLOAT_DISABLE (false)
/** Overflow policy for the tracebuffer */
typedef enum
{
set_overwrite, /**< Overwrite oldest samples */
set_drop /**< Drop newest samples */
} set_overflow_policy_t;
/** Element used to allocate the tracebuffer */
typedef struct
{
char const * m_tag;
uint32_t m_timestamp;
uint32_t m_value;
uint8_t m_type;
uint8_t m_sub;
} set_trace_t;
/** @brief Integration hook writing out a packet to the desired interface
* @param[in] str Packet data
* @param[in] len Packet data length
* @return None
*/
void set_out(const char* const str, size_t len);
/** @brief Integration hook for retrieving a timestamp
* @return Timestamp value
*/
uint32_t set_timestamp(void);
/** @brief Integration hook which locks the tracebuffer
* This can be for instance a mutex or interrupt lock.
* @param[out] h handle value which will be passed to the #set_crit_off function
*/
void set_crit_on(uint32_t *h);
/** @brief Integration hook which locks the tracebuffer
* This can be for instance a mutex or interrupt lock.
* @param[in] h handle value set by #set_crit_on
*/
void set_crit_off(const uint32_t h);
/** @brief Initializes PET with the provided trace buffer
* @param[in] str Trace buffer array
* @param[in] len Trace buffer array length
* @return None
*/
void set_init(set_trace_t tracebuffer[], size_t ntraces);
/** @brief Returns the current trace buffer utilization
* @return Number of items used in the trace buffer
*/
size_t set_get_buffer_items(void);
/** @brief Enables or disables TTY mode
* TTY mode esures packet output is friendly for interactive console usage without a client.
* This means that at most 4 packets are sent at once, the data is encoded as hex digits
* and position is restored with each transmit. To erase a transmitted packet, see set_tty_rubout.
* @param[in] enable If true, enable TTY mode
* @return None
*/
void set_tty_mode(const bool enable);
/** @brief Creates a rubout packet clearing the maximum amount of data transmitted in TTY mode.
* This consists of a ANSI store, enough spaces to overwrite sent data and an ANSI restore.
* @param[out] buffer output buffer to render the rubout packet
* @param[in] buffer_size the size of the output buffer
* @return The packet size written to the buffer
*/
size_t set_tty_rubout(char buffer[], const size_t buffer_size);
/** @brief Enables tracing with the provided overflow policy
* @param[in] policy Either overwrite to overwrite oldest trace, or drop to drop the newest sample
* @return None
*/
void set_enable(const set_overflow_policy_t policy);
/** @brief Disables tracing
* @return None
*/
void set_disable(void);
/** @brief Clears the trace buffer deleting all captures samples
* @return None
*/
void set_clear(void);
/** @brief Render a set amount of tracebuffer items and write them out using set_out
* This function is for basic usage where the output function writes into a managed stream such as stdout
* @param[in] n Number of tracebuffer items to render, or 0 for the current size of the tracebuffer
* @param[in] compress if true, render the packets compressed
* @return None
*/
size_t set_output(size_t n, const bool compress);
/** @brief Render a set amount of tracebuffer items and store them in the provided buffer
* This advanced function is intended to be used with DMA buffers.
* @param[out] buffer output buffer to render the packets
* @param[in] buffer_size the size of the output buffer
* @param[in] n Number of tracebuffer items to render, or 0 for the current size of the tracebuffer
* @param[in] compress if true, render the packets compressed
* @return None
*/
size_t set_render(char buffer[], const size_t buffer_size, const size_t n, const bool compress);
/** @brief Adds diagnostic samples to the tracebuffer
* Signals include:
* - PET.BufferItems:u32 - The current amount of tracebuffer items
* - PET.BufferHealth:u8 - The buffer utilization, zero means that the buffer is full while 255 means the buffer is free
* - PET.CompressionLevel:u8 - The average compression level if packets are rendered compressed
* - PET.CompressionTime:u32 - The average compression time spent for one packet render
* - PET.RenderTime:u32 - The average render time for one packet
* - PET.ItemsSent:u32 - The number of items sent since the last call to this function
* @return None
*/
void set_diagtrace(void);
/** @brief Trace an event
* @param[in] tag Signal name of the trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_evtrace(const char* const tag, const bool skip_time);
/** @brief Trace an u8 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_u8trace(const char* const tag, const uint8_t value, const bool skip_time);
/** @brief Trace an u16 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_u16trace(const char* const tag, const uint16_t value, const bool skip_time);
/** @brief Trace an u32 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_u32trace(const char* const tag, const uint32_t value, const bool skip_time);
/** @brief Trace an s8 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_s8trace(const char* const tag, const int8_t value, const bool skip_time);
/** @brief Trace an s16 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_s16trace(const char* const tag, const int16_t value, const bool skip_time);
/** @brief Trace an s32 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_s32trace(const char* const tag, const int32_t value, const bool skip_time);
/** @brief Trace an array of u8 values
* @param[in] tag Signal name of the trace
* @param[in] value Value array to trace
* @param[in] size Size of the value array, must be an integer literal for the client to detect the signal correctly
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_au8trace(const char* const tag, const uint8_t value[const], const uint8_t size, const bool skip_time);
/** @brief Trace an array of u16 values
* @param[in] tag Signal name of the trace
* @param[in] value Value array to trace
* @param[in] size Size of the value array, must be an integer literal for the client to detect the signal correctly
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_au16trace(const char* const tag, const uint16_t value[const], const uint8_t size, const bool skip_time);
/** @brief Trace an array of u32 values
* @param[in] tag Signal name of the trace
* @param[in] value Value array to trace
* @param[in] size Size of the value array, must be an integer literal for the client to detect the signal correctly
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_au32trace(const char* const tag, const uint32_t value[const], const uint8_t size, const bool skip_time);
/** @brief Trace an array of s8 values
* @param[in] tag Signal name of the trace
* @param[in] value Value array to trace
* @param[in] size Size of the value array, must be an integer literal for the client to detect the signal correctly
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_as8trace(const char* const tag, const int8_t value[const], const uint8_t size, const bool skip_time);
/** @brief Trace an array of s16 values
* @param[in] tag Signal name of the trace
* @param[in] value Value array to trace
* @param[in] size Size of the value array, must be an integer literal for the client to detect the signal correctly
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_as16trace(const char* const tag, const int16_t value[const], const uint8_t size, const bool skip_time);
/** @brief Trace an array of s32 values
* @param[in] tag Signal name of the trace
* @param[in] value Value array to trace
* @param[in] size Size of the value array, must be an integer literal for the client to detect the signal correctly
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_as32trace(const char* const tag, const int32_t value[const], const uint8_t size, const bool skip_time);
/** @brief Trace a string
* @param[in] tag Signal name of the trace
* @param[in] str String to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_strtrace(const char* const tag, const char* const str, const bool skip_time);
#if (!SET_FLOAT_DISABLE)
/** @brief Trace an f32 value
* @param[in] tag Signal name of the trace
* @param[in] value Value to trace
* @param[in] skip_time Use last recorded timestamp for this trace
* @return None
*/
void set_f32trace(const char* const tag, const float value, const bool skip_time);
#endif
#endif

543
set_record.py Normal file
View File

@ -0,0 +1,543 @@
# -*- coding: utf-8 -*-
import sys
import os
import re
import struct
import argparse
import datetime
from functools import partial
from vcd import VCDWriter, writer
from vcd.common import VarType
header = """
▓▓▓▓▓ ░░░ ███████ ███████ ████████
▓ ▓ ▒ ░ ██ ██ ██ Streaming Event Trace
▓ ▓ ▓ ░░ ███████ █████ ██ VCD Recorder Utility
▓ ▓ ▒ ░ ██ ██ ██
▓▓▓▓▓ ░░░ ███████ ███████ ██ (c) 2025 D.Hoeglinger
"""
def human_readable_size(size, decimal_places=2):
for unit in ['B', 'KiB', 'MiB', 'GiB', 'TiB', 'PiB']:
if size < 1024.0 or unit == 'PiB':
break
size /= 1024.0
return f"{size:.{decimal_places}f} {unit}"
def is_hex(s):
return all(chr(c) in '0123456789abcdefABCDEF' for c in s)
def bsd_hash(data):
checksum = 0
for c in data:
checksum = (checksum >> 1) + ((checksum & 1) << 15)
checksum += ord(c)
checksum = checksum & 0xFFFF
return checksum
def hex_to_int(hex_digit):
if hex_digit.isdigit():
return int(hex_digit)
elif hex_digit in 'ABCDEF':
return ord(hex_digit) - ord('A') + 10
elif hex_digit in 'abcdef':
return ord(hex_digit) - ord('a') + 10
else:
raise ValueError("Invalid hexadecimal digit")
def decode_hexstr(payload):
value = bytearray()
for i in range(0, len(payload), 2):
b = hex_to_int(chr(payload[i]))
b |= hex_to_int(chr(payload[i + 1])) << 4
value.append(b)
return bytes(value)
def hexstr(payload):
return payload.hex().upper()
def decode_binstr(payload):
value = 0
for i,b in enumerate(payload):
value |= b << (8 * i)
return value
class DecodeError(Exception):
pass
def cobs_decode(enc, delim):
enc = list(enc)
enc.append(delim)
enc2 = enc[0:-1]
length = len(enc2)
if delim != 0x00:
for i in range(0, len(enc2)):
enc2[i] = enc[i] ^ delim
dec = []
code = 0xFF
block = 0
for i in range(0, length):
byte = enc2[i]
if block != 0:
dec.append(byte)
else:
if (i + byte) > len(enc):
raise DecodeError(f"Marker pointing to end of packet, at {i}, marker={byte}")
if code != 0xFF:
dec.append(0)
code = byte
block = code
if code == 0:
break
block = block - 1
return bytes(dec)
def _memmove(data: bytearray, stidx: int, offset: int, mlen: int) -> None:
for i in range(mlen):
data[stidx + i] = data[stidx - offset + i]
def fastlz_decompress_lv1(datain, doutlen):
opcode_0 = datain[0]
datain_idx = 1
dataout = bytearray(doutlen)
dataout_idx = 0;
while True:
op_type = opcode_0 >> 5
op_data = opcode_0 & 31
if op_type == 0b000:
# literal run
run = 1 + opcode_0
dataout[dataout_idx:dataout_idx + run] = datain[datain_idx:datain_idx + run]
datain_idx += run
dataout_idx += run
elif op_type == 0b111:
# long match
opcode_1 = datain[datain_idx]
datain_idx += 1
opcode_2 = datain[datain_idx]
datain_idx += 1
match_len = 9 + opcode_1
ofs = (op_data << 8) + opcode_2 + 1
_memmove(dataout, dataout_idx, ofs, match_len)
dataout_idx += match_len
else:
# short match
opcode_1 = datain[datain_idx]
datain_idx += 1
match_len = 2 + op_type
ofs = (op_data << 8) + opcode_1 + 1
_memmove(dataout, dataout_idx, ofs, match_len)
dataout_idx += match_len
if datain_idx < len(datain):
opcode_0 = datain[datain_idx]
datain_idx += 1
else:
break
return bytes(dataout[:dataout_idx])
def scan_for_signals(directory, predefined_signals):
library_files = ['pet.c', 'pet.h']
rx_events = re.compile(r'pet_evtrace\(\"([^\"]+)\"')
rx_scalars = re.compile(r'pet_([usf])(8|16|32)trace\(\"([^\"]+)\"')
rx_arrays = re.compile(r'pet_a([us])(8|16|32)trace\(\"([^\"]+)\"\,\s*[^,]+,\s*((?:0x)?[a-zA-Z0-9]+)')
rx_strings = re.compile(r'pet_strtrace\(\"([^\"]+)\"')
signals = {}
valid = True
def add_variable(tag, variable):
hash = bsd_hash(tag)
if hash in signals:
if signals[hash] != variable:
print(f"{file_path}: error: variable `{variable}` has conflicting hash with `{signals[hash]}`")
valid = False
else:
signals[hash] = variable
for predef_signal in predefined_signals:
sig,_ = predef_signal.split(":")
add_variable(sig, predef_signal)
for root, _, files in os.walk(directory):
for file in files:
if file.endswith(('.c', '.h')) and (file not in library_files):
file_path = os.path.join(root, file)
with open(file_path, 'r') as f:
content = f.read()
for match in rx_events.finditer(content):
tag = match.group(1)
variable = f"{tag}:event"
add_variable(tag, variable)
for match in rx_scalars.finditer(content):
tag = match.group(3)
variable = f"{tag}:{match.group(1)}{match.group(2)}"
add_variable(tag, variable)
for match in rx_arrays.finditer(content):
tag = match.group(3)
variable = f"{tag}:{match.group(1)}{match.group(2)}[{int(match.group(4),0)}]"
add_variable(tag, variable)
for match in rx_strings.finditer(content):
tag = match.group(1)
variable = f"{tag}:string"
add_variable(tag, variable)
return list(signals.values()), valid
class Filter:
PREAMBLE = b"\x1b[s"
EPILOUGE = b"\x1b[u"
TAGCODE_LUT = {
0x11 : "D1",
0x12 : "D2",
0x14 : "D4",
0x21 : "V1",
0x22 : "V2",
0x24 : "V4",
0x31 : "A1",
0x32 : "A2",
0x34 : "A4",
0x44 : "S4",
0x54 : "F4",
0x60 : "EV"
}
FLAGS_COMPRESSED = (1 << 0)
def __init__(self, on_value, on_noise):
self.preamble_i = 0
self.epilouge_i = 0
self.packet_buffer = []
self.noise_buffer = []
self.process_value = on_value
self.process_noise = on_noise
self.packet_counter = 0
self.packets_dropped = 0
def process(self, b):
if self.preamble_i == (len(self.PREAMBLE)):
self.packet_buffer.append(b)
if b == self.EPILOUGE[self.epilouge_i]:
self.epilouge_i += 1
else:
self.epilouge_i = 0
if self.epilouge_i == (len(self.EPILOUGE)):
self.process_packet(self.packet_buffer[:-len(self.EPILOUGE)])
self.packet_buffer = []
self.preamble_i = 0
self.epilouge_i = 0
else:
if b == self.PREAMBLE[self.preamble_i]:
self.preamble_i += 1
self.noise_buffer.append(b)
else:
self.preamble_i = 0
for nb in self.noise_buffer:
self.process_noise(nb)
self.noise_buffer = []
self.process_noise(b)
def disassemble_packet(self, packet):
try:
tagcode = packet[0]
# ignore rubout packets
if chr(tagcode) == ' ':
return
if tagcode not in self.TAGCODE_LUT:
self.packets_dropped += 1
print("LUT ERR", tagcode)
return
tag = self.TAGCODE_LUT[tagcode]
value = None
offset = 1
match tag:
case "D1"|"V1"|"A1":
value = decode_binstr(packet[offset:offset+1])
offset += 1
case "D2"|"V2"|"A2":
value = decode_binstr(packet[offset:offset+2])
offset += 2
case "D4"|"V4"|"A4"|"F4"|"S4":
value = decode_binstr(packet[offset:offset+4])
offset += 4
sub = None
if tag[0] == 'A' or tag[0] == 'S':
sub = decode_binstr(packet[offset:offset+1])
offset += 1
hashtag = None
if tag[0] != 'D':
hashtag = decode_binstr(packet[-2:])
except Exception as ex:
self.packets_dropped += 1
return
self.process_value(hashtag, value, sub, tag)
self.packet_counter += 1
def disassemble_macropacket(self, packet):
flags = packet[0]
payload = packet[1:]
try:
macropack = cobs_decode(payload, 0x1B)
if self.FLAGS_COMPRESSED & flags:
frames = fastlz_decompress_lv1(macropack, 1024)
else:
frames = macropack
macropack_delim = 0x00
for pack in frames.split(bytes(bytearray([macropack_delim]))):
if len(pack) == 0:
continue
try:
decoded = cobs_decode(pack, macropack_delim)
self.disassemble_packet(decoded)
except:
self.packets_dropped += 1
except:
self.packets_dropped += 1
def process_packet(self, packet):
if len(packet) == 0:
return
if is_hex(packet):
packet = decode_hexstr(packet)
if True or (len(packet) > 3 + 8 + 3):
self.disassemble_macropacket(packet)
else:
self.disassemble_packet(packet)
class Retagger:
def __init__(self, on_value, tags=None):
tags = tags or []
self._tag_lut = {bsd_hash(tag):tag for tag in tags}
self.process_value = on_value
self.packets_dropped = 0
def process(self, hashtag, number, sub, datatag):
if hashtag in self._tag_lut or datatag[0] == 'D':
tag = self._tag_lut.get(hashtag, None)
self.process_value(tag, number, sub, datatag)
else:
self.packets_dropped += 1
class VcdSink:
def __init__(self, fs, signals, timescale='1 us'):
self.writer = VCDWriter(fs, timescale=timescale, date=datetime.datetime.now().isoformat(), version=f"PET v1.0")
self.skalars = {}
self.arrays = {}
self.strings = {}
self.varnames = {}
self.timestamp = 0
self.packets_dropped = 0
for v in signals:
hvar, vtype = v.split(":")
hier, _, name = hvar.rpartition(".")
arr = None
s = vtype.split("[")
if len(s) == 2:
vtype, arr = s
dsize = 32
dtype = 'integer'
match vtype:
case 'event':
dtype = 'event'
dsize = 32
case 'f32':
dtype = 'real'
dsize = 32
case 'u32'|'s32':
dtype = 'integer'
dsize = 32
case 'u16'|'s16':
dtype = 'integer'
dsize = 16
case 'u8'|'s8':
dtype = 'integer'
dsize = 8
case 'string':
dtype = 'string'
dsize = 8
self.varnames[hvar] = hvar
if arr is not None:
elems = int(arr.rstrip("]"))
vars = []
for i in range(0, elems):
vars.append(self.writer.register_var(hvar, f"{name}[{i}:{(i+1)}]", 'wire', size=dsize))
self.arrays[hvar] = vars
elif dtype == 'string':
self.strings[hvar] = [self.writer.register_var(hier, name, dtype, size=dsize), ""]
else:
self.skalars[hvar] = self.writer.register_var(hier, name, dtype, size=dsize)
def process(self, tag, value, sub, datatag):
if datatag[0] == 'D':
self.timestamp += value
# array values
elif datatag[0] == 'A':
timestamp = self.timestamp
try:
#print(f"### {timestamp:012X} : {self.varnames[tag]}[{sub}] <= {value} [OK] ", flush=True)
self.writer.change(self.arrays[tag][sub], timestamp, value)
except ValueError:
print(f"### {timestamp:012X} : {self.varnames[tag]}[{sub}] <= {value} [VAL_ERR] ", flush=True)
self.packets_dropped += 1
except writer.VCDPhaseError:
print(f"### {timestamp:012X} : {self.varnames[tag]}[{sub}] <= {value} [PHA_ERR] ", flush=True)
self.packets_dropped += 1
except:
print(f"### {timestamp:012X} : {self.varnames[tag]}[{sub}] <= {value} [ERR] ", flush=True)
self.packets_dropped += 1
elif datatag == 'S4':
timestamp = self.timestamp
# unpack
for i in range(0,4):
char = chr(value >> (i*8) & 0xFF)
self.strings[tag][1] += char
# sub of 1 indicates end of string
if sub == 1:
try:
#print(f"### {timestamp:012X} : {self.varnames[tag]} <= \"{self.strings[tag][1]}\"", flush=True)
self.writer.change(self.strings[tag][0], timestamp, self.strings[tag][1])
except ValueError:
print(f"### {timestamp:012X} : {self.varnames[tag]} <= \"{self.strings[tag][1]}\" [VAL_ERR] ", flush=True)
self.packets_dropped += 1
except writer.VCDPhaseError:
print(f"### {timestamp:012X} : {self.varnames[tag]} <= \"{self.strings[tag][1]}\" [PHA_ERR] ", flush=True)
self.packets_dropped += 1
except:
print(f"### {timestamp:012X} : {self.varnames[tag]} <= \"{self.strings[tag][1]}\" [ERR] ", flush=True)
self.packets_dropped += 1
self.strings[tag][1] = ""
# skalar values
elif (datatag == 'EV') or (datatag[0] == 'V') or (datatag[0] == 'F'):
timestamp = self.timestamp
try:
if self.skalars[tag].type == VarType.event:
value = True
elif datatag == 'F4':
value = struct.unpack(">f", struct.pack(">L", value))[0]
#print(f"### {timestamp:012X} : {self.varnames[tag]} <= {value:08X}", flush=True)
self.writer.change(self.skalars[tag], timestamp, value)
except ValueError:
print(f"### {timestamp:012X} : {self.varnames[tag]} <= {value} [VAL_ERR] ", flush=True)
self.packets_dropped += 1
except writer.VCDPhaseError:
print(f"### {timestamp:012X} : {self.varnames[tag]} <= {value} [PHA_ERR] ", flush=True)
self.packets_dropped += 1
except:
print(f"### {timestamp:012X} : {self.varnames[tag]} <= {value} [ERR] ", flush=True)
self.packets_dropped += 1
def process_noise(noisefile, b):
print(chr(b), end="", flush=True)
if noisefile:
noisefile.write(b.to_bytes(1))
def main():
parser = argparse.ArgumentParser(description="scans stdin for PET packets and dumps the values into a VCD file")
parser.add_argument('-d', '--dump', type=str, required=True,
help='output IEEE 1364-2005 Value Change Dump (vcd) file')
parser.add_argument('-t', '--timescale', type=str, default="1 us",
help='period of one timestamp tick')
parser.add_argument('-n', '--noise', type=str, default=None,
help='store the stdin data sans packets in this file')
parser.add_argument('-s', '--source', type=str, required=True,
help='source tree to scan for trace marks')
parser.add_argument('--diagnostics', action=argparse.BooleanOptionalAction,
help='add additional signals tracing internal state of PET')
args = parser.parse_args()
print(header)
tracefile = args.dump
noisefile = args.noise
source_tree = args.source
timescale = args.timescale
enable_diag = args.diagnostics
predefined_signals = []
if enable_diag:
predefined_signals += [
'PET.BufferItems:u32',
'PET.BufferHealth:u8',
'PET.CompressionLevel:u8',
'PET.CompressionTime:u32',
'PET.RenderTime:u32',
'PET.ItemsSent:u32'
]
signals, signals_valid = scan_for_signals(source_tree, predefined_signals)
if not signals_valid:
return
signals.sort()
tags = [k.split(":")[0] for k in signals]
dfile = open(tracefile, 'w', encoding='utf-8')
process_noise_p = partial(process_noise, None)
nfile = None
if noisefile:
nfile = open(noisefile, 'wb')
process_noise_p = partial(process_noise, nfile)
vcd_sink = VcdSink(dfile, signals, timescale)
retagger = Retagger(vcd_sink.process, tags)
packet_filter = Filter(retagger.process, process_noise_p)
print("Signals:")
for var in signals:
print(f" - {var}")
print()
print(" === BEGIN NOISE ===")
try:
while True:
for b in sys.stdin.buffer.read(1):
packet_filter.process(b)
except KeyboardInterrupt:
pass
print()
print(" === END NOISE ===")
print()
vcd_sink.writer.close()
dfile.close()
if nfile:
nfile.close()
print("Summary:")
packet_count = packet_filter.packet_counter
drop_count = packet_filter.packets_dropped + vcd_sink.packets_dropped
trace_size = human_readable_size(os.path.getsize(tracefile))
print(f" - Packets received: {packet_count}")
print(f" - Packets dropped: {drop_count}")
print(f" - Trace file: {tracefile}")
print(f" - Trace size: {trace_size}")
if __name__ == '__main__':
main()