Reading Traces

As a comprehensive system, traces in Neutrino is not only a file but a system.

Trace Folder Structure

NEUTRINO's traces will be placed in folders, commonly like trace/Apr24_231539_1860576/, of:

• A parent folder, default to be ./trace and is configurable via --tracedir option of NEUTRINO CLI.
• Traces of each process or run will be placed in a subfolder whose name follows: <DATE>_<TIME>_<PID>. For example, Apr24_231539_1860576 means a trace recorded at 23:15:39 of April 24 (local timezone) from process 1860576.

Each trace folder are commonly structured as:

At minimum, there are two files, event.log of Hook Driver internal logs for troubleshooting and probe.toml of compiled assembly probes for reproducing experiments. There are also two folders, result containing .bins of trace results and kernel contains kernel captured and jitted.

Trace Structure

Each trace (.bin in result/) is structured in ELF-like manner, starting with a header, several sections (each for a map), and raw binaries whose offset corresponds to section definitions:

typedef struct {
    uint32_t gridDimX;
    uint32_t gridDimY;
    uint32_t gridDimZ;
    uint32_t blockDimX;
    uint32_t blockDimY;
    uint32_t blockDimZ;
    uint32_t sharedMemBytes;
    uint32_t numMaps; // number of traces exposed
} trace_header_t;

typedef struct {
    uint64_t size;   // size of record per thread/warp in bytes
    uint64_t offset; // offset for fseek
} trace_section_t;

By fseek to section.offset and fread(gridSize * blockSize * section.size), one can safely read out the traces in CPU for analysis.

Event Log Structure

event.log is the operation logs of the Hook Driver, and many functionalities, such as measuring probe overhead or troubleshooting, relies on parsing this log.

Kernel Folder Structure

Each folder inside kernel corresponds to a kernel, named by an auto-incremental index and the SHA1 of kernel name (since C++ template demangled name can easily exceeds Linux folder name limitation). Within each folder, there are:

• process.log: the operation log of the Probe Engine.
• original.bin and original.asm: the raw machine code and assembly dumped by the Hook Driver. Be aware, they can be large!
• pruned.bin and pruned.asm: the pruned machine code and assembly to understand the original code.
• probed.bin and probed.asm: the probed machine code (to be loaded back and execute) and assembly for troubleshooting.