I’ve built what I think is a pretty neat Ruby tracing profiler. It captures four event streams: Ruby function calls, system calls, thread-state changes, and garbage-collection activity, while adding less than 30 ns of overhead per Ruby function call, low enough for continuous use in large-scale production systems.

As far as I know, no other Ruby tracer offers this mix of signals at this cost. If you’re aware of one, please let me know and I’ll add a note here (and remove a couple of adjectives from the title).

I haven’t open-sourced the code yet because it is very much in a proof-of-concept state and I’m busy with other projects. If you think it would be valuable to you, let me know. If I see enough interest I’ll move it up my priority list.

Overhead Measurement

Above is a runtime histogram of 200 runs of a sample workload on an i5-13500 system running Ubuntu 24 compared to baseline Ruby and ruby-prof 1.7.1, the profiler that comes closest in terms of overhead. Estimated per-function-call overhead is 23 ns for this profiler and 538 ns for ruby-prof.

Why Trace in Production

In a word: outliers. Sampling profilers show where a program spends its time on average. However, issues like high tail latency are, by definition, poorly represented in an average view. That’s where a tracing profiler comes in handy.

Let me illustrate this with a sophisticated SaaS simulator that handles a bunch of API calls and records the duration of each one:

durations = []

100_000.times do
  start_time = Time.now

  api_handler

  durations << Time.now - start_time
end

Let’s assume api_handler is a black-box function with a P99.9 latency SLO of < 10ms. Running the snippet yields:

P50 latency:     1.1 ms
P99.9 latency: 101.3 ms

Our median (P50) looks fine, but the P99.9 violates the SLO.

If you care about latency, you’re probably already running a sampling profiler, so let’s see what it shows:

We see that two methods account for most of the runtime: api_handler->Object#a and api_handler->Object#b. Of those, a consumes the majority of the time slice. You might therefore conclude that optimizing a is the right move, but in this case, that would be a waste of effort!

To demonstrate this, let’s record a trace from one of the slow api_handler executions using our new profiler. First, we instrument the code:

+ require 'tracing'

 durations = []
 SLA = 0.01

 100_000.times do
   start_time = Time.now
+   Trace.reset

   api_handler

   duration = Time.now - start_time
+  if duration > SLA
+     Trace.save("api_call_trace.json")
+   end

   durations << duration
end

Now let’s look at one of the slow traces:

Here we see the exact opposite: b now consumes the majority of the time slice. So who’s right?

Let’s peek inside the api_handler black-box:

# api_handler always executes a, which sleeps for 1 ms
# 0.1% of the time it also executes b, which sleeps for 100 ms
def api_handler
  a
  if rand >= 0.999
    b
  end
end

def a
  sleep(0.001)
end

def b
  sleep(0.1)
end

The sampling profiler correctly shows that over the entire run, we spend more time in a, but when chasing tail-latency issues, that is not what matters. Here, a tracing profiler is the right tool for the job and b is the right function to optimize.

One small additional advantage of tracing is that it gives us a true time axis: events actually occur in the order shown, unlike in a sampled flame graph. This can be quite helpful when debugging.

Thread-State Changes

Another event stream I’ll mention here is thread-state changes and why tracking them helps. You might assume that whenever you see a stack trace in a trace view, like the one outlined by the red box below, the code is actively executing.

But that isn’t necessarily the case. Notice the light-green Runnable slice and the dark-green Running slice in the example above. That track shows that the Ruby thread was not running at first, later became runnable, and finally began executing. In this case the reason is fairly clear when you inspect the stack trace: the thread is in sleep, waiting on a futex, and in the yellow-and-purple CPU 12 track you can see the Swapper process running on the core instead.

But less obvious instances exist as well: if the CPU you’re running on is oversubscribed, the kernel scheduler may boop your thread off its core as it tries to give other threads time to run. If a code you own starts running slowly because someone else is stealing your cycles, you want to know!

Now, you might already know strace lets you trace system calls, but did you know it can also change their behavior? You can modify their input and output, inject errors and add time delays (though be aware of the limitations).

To demonstrate how it could be useful, I’ve added the ability to easily use this functionality from Python and Ruby to Cirron (my grab-bag of a project, that can also trace syscalls and track performance counters). Now you can do things like:

Test how code handles insufficient space. (I’ll use Python for demonstration purposes here, check out the readme for examples of how to do this in Ruby.)

from cirron import Injector

injector = Injector()
# Make the "openat" syscall return the ENOSPC error.
injector.inject("openat", "error", "ENOSPC")

# All "openat" calls will return ENOSPC within this context.
with injector:
    # Fails with "No space left on device".
    f = open("test.txt", "w")

# From here on "openat" behaves normally again.

Inject occasional errors and delays to network operations.

(...)
# Make every other "connect" syscall return the ETIMEDOUT error.
# when="2+2" means to perform the injection for the second syscall
# invocation and then again every two invocations.
injector.inject("connect", "error", "ETIMEDOUT", when="2+2")

# Also add 1s of latency to "send".
injector.inject("send", "delay_exit", "1s")

with injector:
    (...)

Simulate signals being sent before particular syscalls.

(...)
# Simulate user pressing Ctrl+C before the first "read".
injector.inject("read", "signal", "SIGINT", "when=1") 

with injector:
    (...)

And more! In addition to the error, delay_exit and signal actions demonstrated above, it also supports retval for changing a return value without making it an error, delay_enter for delaying entry into a syscall rather than an exit and poke_enter and poke_exit for modifying the process memory on syscall entry or exit. See the “Tampering” section of strace’s man page for details on all these, including a description of the format of the when argument.

If you want to try this but are unsure what syscalls your code uses you can get a list of them easily with Cirron too:

from cirron import Tracer

# Tracer records all syscalls made within the context.
with Tracer() as t:
    print("Hello!")

print(t)
# [Syscall(name='write', args='1, "Hello!\\n", 7', retval='7', duration='0.000197', timestamp='1725900869.238673', pid='438862')]

How?

Cirron implements the Injector in the simplest way possible: it executes strace with the appropriate inject options and points it to the current process. After leaving the injection context the strace process is killed. If you were to make heavy use of this, it’s probably worth implementing the functionality directly, rather than executing strace every time (let me know if you’d find it useful if Cirron did this more efficiently).

So how does strace do this? Ptrace! It attaches to (or seizes; dramatic!) a process with ptrace(PTRACE_ATTACH, ...). This causes the traced process to stop on (among other things) entry and exit from syscalls. Strace can then inspect and modify the program before letting it continue.

To inject a delay, as with delay_enter or delay_exit, strace simply waits before continuing the process.

To modify the syscall’s inputs or output it uses either PTRACE_POKEDATA (or process_vm_writev) to mess with the traced process’s memory (poke_enter, poke_exit) or PTRACE_POKEUSER to modify the USER area, containing the process’s registers state, which lets you, for example, change the return value (error, retval).

Limitations

There’s the obvious performance impact, particularly if simply using strace instead of using ptrace directly.

Also consider that making a syscall fail this way does not remove the side effects it might have: making a “write” call return an error will still (possibly) perform the “write”, it will just appear to have failed to the application. Similarly, delay injections occur either before syscall entry or after exit, which may impact the program differently compared to introducing a delay during the syscall itself.

“Summing ASCII Encoded Integers on Haswell at the Speed of memcpy” turned out more popular than I expected, which inspired me to take on another challenge on HighLoad: Counting uint8s. I’m currently only #13 on the leaderboard, ~7% behind #1, but I have already learned some interesting things. In this post I’ll describe my complete solution (skip to that) including a surprising memory read pattern that achieves up to ~30% higher transfer rates on fully memory bound, single core workloads compared to naive sequential access, while apparently not being widely known (skip to that).

As before, the program is tuned to the input spec and the HighLoad system: Intel Xeon E3-1271 v3 @ 3.60GHz, 512MB RAM, Ubuntu 20.04. It only uses AVX2, no AVX512.

The Challenge

“Print the number of bytes whose value equals 127 in a 250MB file full of bytes uniformly sampled from [0, 255] sent to standard input.”

Nothing much to it! The solution presented here is ~550x faster than the following naive program.

uint64_t count = 0;
for (uint8_t v; std::cin >> v;) {
    if (v == 127) {
        ++count;
    }
}

std::cout << count << std::endl;
return 0;

The Kernel

You’ll find the full source code of the solution at the end of the post. But first I’ll build up to how it works. The kernel is just three instructions long, so I went straight to an __asm__ block (sorry!).

; rax is the base of the input
; rsi is an  offset to current chunk
vmovntdqa    (%rax, %rsi, 1), %ymm4
; ymm2 is a vector full of 127
vpcmpeqb     %ymm4, %ymm2, %ymm4
; ymm6 is an accumulator, whose bytes
; represent a running count of 127s
; at that position in the input chunk
vpsubb       %ymm4, %ymm6, %ymm6

With this, we iterate over 32-byte chunks of the input and:

• Load the chunk with vmovntdqa (it’s a non-temporal move just for style points, it doesn’t make a difference to runtime).
• Compare each byte in the chunk with 127 using vpcmpeqb, giving us back 0xFF (aka -1) where the byte is equal to 127 and 0x00 elsewhere. For example [125, 126, 127, 128, ...] becomes [0, 0, -1, 0, ...].
• Subtract the result of the comparison from an accumulator. Continuing the example above and assuming a zeroed accumulator, we’d get [0, 0, 1, 0, ...].

Then, to prevent this narrow accumulator from overflowing, we dump it into a wider one every once in a while with the following:

; ymm1 is a zero vector
; ymm6 is the narrow accumulator
vpsadbw      %ymm1,%ymm6,%ymm6
; ymm3 is a wide accumulator
vpaddq       %ymm3,%ymm6,%ymm3

vpsadbw sums every eight bytes in the accumulator together into four 64-bit numbers, then vpadddq sums it with a wider accumulator, that we know won’t overflow and that we extract at the end to arrive at the final count.

So far nothing revolutionary. In fact you can find this kind of approach on StackOverflow: How to count character occurrences using SIMD.

The Magic Sauce

The thing with this challenge is, we do so little computation it’s significantly memory bound. I was reading through the typo-ridden Intel Optimization Manual looking for anything memory related when, on page 788, I encountered a description of the 4 hardware prefetchers. Three of them seemed to help purely with sequential access (what I was already doing), but one, the “Streamer”, had an interesting nuance:

“Detects and maintains up to 32 streams of data accesses. For each 4K byte page, you can maintain one forward and one backward stream can be maintained.”

“For each 4K byte page”. Can you see where this is going? Instead of processing the whole input sequentially, we’ll interleave the processing of successive 4K pages. In this particular case interleaving 8 pages seems to be the optimum. We also unroll the kernel a bit and process a whole cache line (2x32 bytes) in each block.

#define BLOCK(offset) \
    "vmovntdqa    " #offset " * 4096 (%6, %2, 1), %4\n\t" \
    "vpcmpeqb     %4, %7, %4\n\t" \
    "vmovntdqa    " #offset " * 4096 + 0x20 (%6, %2, 1), %3\n\t" \
    "vpcmpeqb     %3, %7, %3\n\t" \
    "vpsubb       %4, %0, %0\n\t" \
    "vpsubb       %3, %1, %1\n\t" \

We put 8 of these inside the main loop, with the offset set to 0 through 7.

This improves the score on HighLoad by some 15%, but if your kernel is even more memory bound, let’s say you just vpaddb the bytes to find their sum modulo 255, you can get up to 30% gain with this. Pretty cool for such a simple change!

Anyway, one other small thing: we add a prefetch for 4 cache lines ahead:

#define BLOCK(offset) \
    "vmovntdqa    " #offset " * 4096 (%6, %2, 1), %4\n\t" \
    "vpcmpeqb     %4, %7, %4\n\t" \
    "vmovntdqa    " #offset " * 4096 + 0x20 (%6, %2, 1), %3\n\t" \
    "vpcmpeqb     %3, %7, %3\n\t" \
    "vpsubb       %4, %0, %0\n\t" \
    "vpsubb       %3, %1, %1\n\t" \
    "prefetcht0   " #offset " * 4096 + 4 * 64 (%6, %2, 1)\n\t"

Why 4 lines ahead? I don’t have a good explanation for that, it’s simply what performs best. Below is a plot of runtime of the solution with prefetch strides from 0 to 100 on a different system (hence why the optimum is elsewhere here). As you can see the curve is quite complex.

The Source

#include <iostream>
#include <cstdint>
#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <immintrin.h>
#include <cassert>

#define BLOCK_COUNT 8
#define PAGE_SIZE 4096
#define TARGET_BYTE 127

#define BLOCKS_8 \
    BLOCK(0)  BLOCK(1)  BLOCK(2)  BLOCK(3) \
    BLOCK(4)  BLOCK(5)  BLOCK(6)  BLOCK(7)

#define BLOCK(offset) \
    "vmovntdqa    " #offset "*4096(%6,%2,1),%4\n\t" \
    "vpcmpeqb     %4,%7,%4\n\t" \
    "vmovntdqa    " #offset "*4096+0x20(%6,%2,1),%3\n\t" \
    "vpcmpeqb     %3,%7,%3\n\t" \
    "vpsubb       %4,%0,%0\n\t" \
    "vpsubb       %3,%1,%1\n\t" \
    "prefetcht0   " #offset "*4096+4*64(%6,%2,1)\n\t"


static inline
__m256i hsum_epu8_epu64(__m256i v) {
    return _mm256_sad_epu8(v, _mm256_setzero_si256());
}

int main() {
    struct stat sb;
    assert(fstat(STDIN_FILENO, &sb) != -1);
    size_t length = sb.st_size;

    char* start = static_cast<char*>(mmap(nullptr, length, PROT_READ, MAP_PRIVATE | MAP_POPULATE, STDIN_FILENO, 0));
    assert(start != MAP_FAILED);

    uint64_t count = 0;
    __m256i sum64 = _mm256_setzero_si256();
    size_t offset = 0;

    __m256i compare_value = _mm256_set1_epi8(TARGET_BYTE);
    __m256i acc1 = _mm256_set1_epi8(0);
    __m256i acc2 = _mm256_set1_epi8(0);
    __m256i temp1, temp2;

    while (offset + BLOCK_COUNT*PAGE_SIZE <= length) {
        int batch = PAGE_SIZE / 64;
        asm volatile(
            ".align 16\n\t"
            "0:\n\t"

            BLOCKS_8

            "add          $0x40, %2\n\t"
            "dec          %5\n\t"
            "jg           0b"
            : "+x" (acc1), "+x" (acc2), "+r" (offset), "+x" (temp1), "+x" (temp2), "+r" (batch)
            : "r" (start), "x" (compare_value)
            : "cc", "memory"
        );

        offset += (BLOCK_COUNT - 1)*PAGE_SIZE;

        sum64 = _mm256_add_epi64(sum64, hsum_epu8_epu64(acc1));
        sum64 = _mm256_add_epi64(sum64, hsum_epu8_epu64(acc2));

        acc1 = _mm256_set1_epi8(0);
        acc2 = _mm256_set1_epi8(0);
    }

    sum64 = _mm256_add_epi64(sum64, hsum_epu8_epu64(acc1));
    sum64 = _mm256_add_epi64(sum64, hsum_epu8_epu64(acc2));

    count += _mm256_extract_epi64(sum64, 0);
    count += _mm256_extract_epi64(sum64, 1);
    count += _mm256_extract_epi64(sum64, 2);
    count += _mm256_extract_epi64(sum64, 3);

    for (; offset < length; ++offset) {
        if (start[offset] == TARGET_BYTE) {
            ++count;
        }
    }

    std::cout << count << std::endl;
    return 0;
}

Conclusion

The page-interleaved read pattern seems surprisingly under-discussed and I don’t remember ever seeing it used in code in the wild. Curious! If you’re aware of it being used anywhere, let me know, I’d love to see it! And if I’m missing any other memory-based optimization, let me know too. :)

“Print the sum of 50 million ASCII-encoded integers uniformly sampled from [0, 2³¹−1], separated by a single new line and sent to standard input.”

On the surface, a trivial problem. But what if you wanted to go as fast as possible?

I’m currently one of the top ranked competitors in exactly that kind of challenge and in this post I’ll show you a sketch of my best performing solution. I’ll leave out some of the µoptimizations and look-up table generation to keep this post short, easier to understand and to not completely obliterate the HighLoad leaderboard. Still, as far as I know nothing similar has been published yet, so I’m hoping you’ll find it interesting.

On the target system, my program runs about 320x faster than the following naive C++ solution (and is about 1,000,000x more fragile):

uint64_t sum = 0;
while (std::cin) {
    uint64_t v = 0;
    std::cin >> v;
    sum += v;
}

std::cout << sum << std::endl;

I’ll write a companion post later on where I’ll describe one of the techniques used here in more detail: what I think is a fairly novel, though certainly very insane, way of initializing sparse, very-wide, zero-overhead lookup tables. The whole story of how I made it work is a little long to fit into this post.

Limitations

The program is over-fit to the input spec and the particular host it runs on (Intel Xeon E3-1271 v3 @ 3.60GHz, 512MB RAM, Ubuntu 20.04). Given the CPU, it only uses SIMD instructions up to AVX2, no AVX512. It assumes the input is exactly according to the spec and hence does zero error handling and even on such input will only produce correct results with probability < 1, though very close to 1, depending on the parameters you choose.

The Algorithm

Here’s the high-level overview: forget about parsing the input number-by-number and keeping a running sum! We’ll instead iterate over 32 byte chunks of the input using SIMD, from back to front, keeping track of the sum of the digits in each decimal place. In other words, if the input was “123\n45\n678”, we’ll remember that we’ve seen a total of 1 + 6 = 7 in the “hundreds”, 2 + 4 + 7 = 13 in the “tens” and 3 + 5 + 8 = 16 in the “ones” place. After we’re done processing the whole input, we get the final sum by multiplying these decimal place sums with powers of ten: 7*10² + 13*10¹ + 16*10⁰ = 846. Note that since the highest number we have to deal with is 2³¹−1, we have to track at most ⌈log₁₀(2³¹−1)⌉ = 10 decimal place sums.

How do we identify which byte of our input chunk is which decimal place? A look-up table. The mapping from the byte of an input chunk to its decimal place is determined by just two things: the location of newlines in the chunk and the length of the leftmost number in the previous chunk. In other words in a chunk like “???\n??\n???” that follows “???\n???\n???”, the first byte is always the 3rd decimal place, then the 2nd, etc., and the last byte is always the 4th decimal place, because it follows a number with 3 digits in the previous chunk.

That’s the high level, but of course the details of the implementation matter a lot too, so let’s look at the source code. This is the meat of the post, I’ve commented the code extensively to explain how it works and why it works that way. It might be hard to read on a phone, in which case I recommend bookmarking it and reading it on desktop later.

// First, the variables and constants we'll need:

// Pointer to the beginning of our input.
char* start = (...)

// Offset from `start` to the first byte of the current 32 byte chunk.
uint64_t offset = (...)

// SIMD vector full of ASCII '\n'.
const __m256i ascii_zero = _m256_set1_epi8(0x30);

// The size of the leftmost number in the previous chunk (remember, we iterate
// input back-to-front), which partially determines the decimal places of the
// rightmost number in the current chunk. Since we iterate from the end of the
// input, we can initialize this to 0.
last_number_size = 0;

// The decimal place sums we use to reconstruct the final sum as described in
// the overview.
uint64_t decimal_sums[10] = {0};

// For efficiency, we accumulate decimal place sums into this vector and dump
// them into the `decimal_sums` array every `BATCH_SIZE` iterations.
// The layout of this vector is below, where a number represents an exponent
// of the power of ten the byte represents:
// [5, 4, 3, 2, 1, 0, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0 | 5, 4, 3, 2, 1, 0, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
//                 ^ this byte accumulates 10^0 = "ones"      ^ this byte accumulates 10^2 = "hundreds"
// The somewhat unusual layout is motivated by the fact that AVX2 shuffle
// cannot move bytes across a lane boundary and because you expect to see
// more low decimal place digits.
__m256i sums_acc = _mm256_set1_epi8(0)

// The decimal place sums accumulator can only accumulate so many chunks
// before overflowing. Worst case scenario is a '9' hitting the same
// accumulator slot twice per iteration of the main loop. Therefore the
// maximum safe accumulation batch size is 255 / (2 * 9) = 14. In practice
// you can increase it to almost twice that number without lowering the
// probability of correct output too much (at least for HighLoad).
const BATCH_SIZE = 14;
int batch = BATCH_SIZE;

// The final sum!
uint64_t sum = 0;

// Next, we ensure our chunks are 64 byte aligned (`start` is already
// aligned) by processing input byte-by-byte with a simple
// scalar algorithm until we reach an aligned offset, keeping track
// of the running sum and the length of the last number encountered.
int exp = 1;
while ((offset & 0xFF) != 0) {
    uint8_t byte = *reinterpret_cast<uint8_t*>(start + offset);
    if (byte == 0x0A) {
        exp = 1;
        last_number_size = 0;
    } else {
        sum += exp * (byte - 0x30);
        exp *= 10;
        last_number_size++;
    }

    offset -= 1;
}

(...)

// Now for the performance critical section!
while (offset) {
    // Prefetch input for future iterations, 11 cache lines forward.
    // 11 chosen empirically.
    _mm_prefetch(reinterpret_cast<const char*>(start + offset - 11*64), _MM_HINT_T0);

    // Load a 32 byte chunk of input.
    __m256i input = _mm256_load_si256(reinterpret_cast<__m256i*>(start + offset));

    // Subtract value of ASCII '0' (0x30) from each byte of the chunk.
    // This accomplishes two things:
    //  1) Bytes that represent digits will now hold the digit value instead
    //     of the ASCII code for the digit. i.e. '0' (value 0x30) will now be
    //     0x00, '1' (value 0x31) will now be 0x01, etc.
    //  2) Bytes that represent newlines will have their top bit set, because
    //     newlines are 0x0A, 0x0A - 0x30 = 0b11011010. This will become
    //     relevant in the next step.
    input = _mm256_sub_epi8(input, ascii_zero);

    // Create a bitmask of newlines in the input chunk, i.e. given a chunk
    // "123\n456\n" return 0b00010001.
    // The bitmask is 32 bits, since the input is 32 bytes and we'll store
    // it in a 64 bit variable with the top half zeroed since we'll need
    // it in a 64 bit context later on when calculating our look-up table
    // location (this generates better assembly).
    // This is where 2) from the comment above comes into play. You might
    // be tempted to say we need `cmpeq(input, 0x0A)` before this `movemask`,
    // but `movemask` only looks at the top bit of each byte to
    // decide the value of the bit in the bitmask, so the fact that the `sub`
    // sets the top bit for each newline, but not for digits, is sufficient.
    uint64_t mask = (uint32_t)_mm256_movemask_epi8(input);

    // The location of the mappings from input bytes to decimal places is
    // determined by the mask and the leftmost number in the previous chunk.
    // There are two mappings for each (mask, last_number_size) pair, 32 bytes
    // each, for one cache line in total and there are 11 of them per newline
    // mask, because last_number_size can be 0 to 10.
    // Here you might be tempted to say we should modify our look-up table
    // structure to let us compute the location as
    // !16! * 64 * mask + 64 * last_number_size.
    // Since we'd only multiply by powers of two, this does result in nicer
    // assembly: `imul` (latency 3), `shl` (latency 1) turns into `shl`, `shl`,
    // but it's ultimately much slower due to cache associativity
    // issues (nice overview of the problem is for example here:
    // https://en.algorithmica.org/hpc/cpu-cache/associativity/)
    uint64_t lut_idx = 11 * 64 * mask + 64 * last_number_size

    // Now we dereference the location to get the two mappings.
    // This is a point where you should be a little confused:
    // 1) We're dereferencing the index by itself, not as an offset into an
    //    array base.
    // 2) The total range of the lut_index variable is very roughly 0 to 2^42,
    //   42 bits address space or some 4TB. That's more than the 500MB RAM we
    //   have available and more than you could create with a normal array
    //   literal.
    // On the positive side the lookup table is very sparse: thanks to the
    // specific input distribution, we only have O(1,000s) chunks of mappings
    // spread over the whole 42 bit address space.
    // This is the part that I'll explain in more detail in a follow up post,
    // but for now you can imagine as if we `mmap` and `memcpy` all the mappings
    // to the right addresses at the start of the program. Let me know if you
    // think you know how to do it ~zero-cost! :)
    // Before I came up with this approach I used a derived index based on the
    // size of the numbers in the chunk using chained `tzcnt`. This shrinks the
    // index space to (barely) fit in a normal look up table, but the index
    // computation was one long dependency chain with high latency instructions
    // and ended up being almost 50% of the runtime.
    __m256i shuffle_ctrl1 = _mm256_loadu_si256(reinterpret_cast<__m256i*>(lut_idx));
    __m256i shuffle_ctrl2 = _mm256_loadu_si256(reinterpret_cast<__m256i*>(lut_idx + 32));

    // When I talked about a mapping from input bytes to decimal places, it's
    // really a shuffle control mask that moves input bytes into the same
    // layout that the `sums_acc` vector has.
    // This is another one of those places where the specific input distribution
    // really matters:
    // There are 4 spots for "ones" in `sums_acc`. If our input chunk consisted
    // of only 1 digit numbers, "1\n2\n\3...", we'd need to perform the following
    // `shuffle`, `add` procedure up to 32 / 2 / 4 = 4 times.
    // Fortunately for us, this turns out to be very unlikely and we are almost
    // guaranteed to be able to completely accumulate the chunk within
    // 2 `shuffles`, so that is what we do in exchange for occasionally
    // producing incorrect results.
    // One of my solutions had a neat compression scheme here:
    // AVX2 `shuffle` only shuffles within each 16 byte lane of the full 32 byte
    // register. It therefore only uses the lower 4 bits of the shuffle control
    // mask so you can trivially pack two of them into one 32 byte vector.
    // Sort of -- it also uses the top bit to let you zero out a byte, which
    // we use a fair bit, (as you can imagine, by the second shuffle we've
    // accumulated most of the bytes of the input). Fortunately,
    // we are guaranteed to have at least one newline in each lane and since
    // we do not care about its value, we can zero it out at the start of each
    // iteration and point bytes that should to be zero to it, rather than
    // zeroing them out using the top bit.
    // I was very excited when I came up with this, but it turned out not
    // to do much performance-wise :) (Haswell can handle two loads per
    // cycle and the shuffle control maps are on a single cache line.)
    __m256i shuffled_input1 = _mm256_shuffle_epi8(input, shuffle_ctrl1);
    __m256i shuffled_input2 = _mm256_shuffle_epi8(input, shuffle_ctrl2);

    // Shuffled inputs 1 & 2 are now in the correct layout for us to add them
    // directly to the decimal place sums accumulator.
    sums_acc = _m256_add_epi8(sums_acc, shuffled_input1);
    sums_acc = _m256_add_epi8(sums_acc, shuffled_input2);

    // This stores the size of the leftmost number for the next iteration.
    // Note that on Haswell this will generate `xor B, B` in addition to
    // `tzcnt A, B`. This is meant as a fix for a false dependency bug on
    // bunch of BMI instructions on this µarch.
    // In our case the fix is counterproductive because we're not bottlenecked
    // on the latency of this instruction. I don't know of any way to bypass
    // that `xor` other than using __asm__ directly.
    // (More on the bug:
    // https://stackoverflow.com/questions/25078285/replacing-a-32-bit-loop-counter-with-64-bit-introduces-crazy-performance-deviati)
    last_number_size = _tzcnt_u32(mask);

    // Once we accumulate `BATCH_SIZE` chunks in `sums_acc`, dump them into
    // the sums array.
    batch--;
    if (!batch) {
        batch = BATCH_SIZE;
        // Extract all accumulated "ones"...
        decimal_sums[0] += _m256_extract_epi8(sums_acc, 5);
        decimal_sums[0] += _m256_extract_epi8(sums_acc, 15);
        decimal_sums[0] += _m256_extract_epi8(sums_acc, 21);
        decimal_sums[0] += _m256_extract_epi8(sums_acc, 31);
        (...)
        // ...and up to 10^9's.
        decimal_sums[9] += _m256_extract_epi8(sums_acc, 6);
        decimal_sums[9] += _m256_extract_epi8(sums_acc, 22);
    }

    // Move the offset to the next batch.
    offset -= 32;
}

(...)

// All that's left to do is to multiply our accumulated decimal place sums by
// the right power of ten and sum to get the final sum.
exp = 1;
for (int i = 0; i < 10; i++) {
    sum += decimal_sums[i] * exp;
    exp *= 10;
}

// And there it is!
print(sum)

Fin

Let me know if you have any feedback and thank you to the HighLoad community on Telegram, especially gracefu and Jack Frigaard.

To put it through its paces I’ve compared what syscalls each language uses for several common patterns: File IO, generating random numbers, telling time and even just printing a string.

File IO

Let’s start with something surprising right away. Here are the snippets under investigation, simply writing a string to a file (I’ll be omitting the Cirron setup from the snippets later on):

# python
with Cirron.Tracer() as t:
    with open("test.txt", "w") as f:
        f.write("Hello, File I/O!")

# ruby
t = Cirron::tracer do
  File.open("test.txt", "w") do |f|
    f.write("Hello, File I/O!")
  end
end

And here then, are the system calls each language makes. I’ve highlighted calls that aren’t made by the other language and added approximate time spent in each call on my system:

Both start off with openat to open test.txt in the current directory. Python then, unlike Ruby, uses newfstat to figure out what buffer size to use by looking at st_blksize¹. Both languages then use ioctl to figure out whether the file descriptor is a TTY.

So far as expected, but then… Python decides to lseek to the start of the file twice. This doesn’t seem strictly necessary and a quick look through the cPython codebase didn’t reveal why this is happening. If you know, let me know!

Anyway: finally both languages use write to write our string and close to close the file descriptor.

How about reading a file?

# python
with open("test.txt", "r") as f:
    content = f.read()

# ruby
content = File.read("test.txt")

Nothing terribly surprising here. Python still doing its double lseek and Ruby, probably not wanting to be left behind, adds an lseek of its own (also not strictly necessary, as far as I can tell). Note how openat here takes only ~30µs, but it was ~80µs when opening for with O_WRONLY|O_CREAT|O_TRUNC.

Sneaky syscalls

Let’s look at generating random numbers and telling time:

# python
random_int = random.randint(1, 100)

# ruby
random_int = rand(1..100)

# python
current_time = time.time()

# ruby
current_time = Time.now

You might reasonably assume you’d need a system call here too, but not so. Since these operations are frequently used and not privileged, they use vDSO and turn into normal function calls, avoiding paying the price of a syscall. This makes them invisible to strace but you’d see them pop up in ltrace.

Printing

Here’s one I thought would be obvious, but turned out a little surprising too:

# python
print("Hello")

# ruby
puts "Hello"

I expected both languages to just use write, but apparently some 6 years ago Ruby switched to using writev instead in Feature #14042: IO#puts: use writev if available. Before this, puts would use an extra write to output a newline and hence wasn’t atomic.

Fin

I hope you found this interesting! If you’d find it useful if Cirron supported another language too, let me know!

I’ll show you a little tiny tool I added to Cirron that lets you see exactly what syscalls a piece of Python code is calling and how to analyze the trace more effectively.

Let’s start with print("Hello") as before:

from cirron import Tracer

with Tracer() as t
    print("Hello")

print(t.trace)
# write(1, "Hello\n", 6) = 6 <0.000150s>

You can see¹ print uses only a single write to write the string "Hello\n" to stdout (that’s what the 1 stands for) and asks it to write at most 6 bytes. Write then returns 6, meaning it managed to write all the bytes we asked it to. You can also see it took 0.00015s or 150μs (that’s just the write call, not the whole print statement).

Pretty cool!

How does Tracer work? I initially wanted to use the ptrace syscall to implement it, but that turned out to be a little more complicated that what I wanted, so in the end I just used the strace tool, which also uses ptrace but handles all the complexity. Tracer simply starts tracing itself with it, redirecting output to a file, which is then parsed when it’s asked to stop.

Let’s trace import seaborn now:

from cirron import Tracer

with Tracer() as t:
    import seaborn

print(len(t.trace))
# 20462

Turns out importing Seaborn takes ~20k syscalls! That’s obviously too many to just print out, so what’s a better way to analyze what it’s doing?

Visualizing traces with Perfetto

Perfetto Trace Viewer let’s you visualize all kinds of traces. It can’t ingest strace output directly, but I’ve included a function that converts Tracer output to Trace Event Format, something Perfetto can load:

from cirron import to_tef

(...)

open("/tmp/trace", "w").write(to_tef(t.trace))

This gets you a file you can open with Perfetto. I’m not going to describe all it can do; I uploaded a trace of import seaborn here, go play with it!

I was surprised to find it uses 4 threads, which mostly spend time looking up files and reading them, but one of the threads seems to be very curious about your CPU details!

Backlinks

Discussions on HackerNews and /r/Python.

In this TT talks about how mathematicians usually work by themselves or with colleagues they trust a lot, because otherwise you spend a lot of time checking everyone’s work for errors.

With Lean (a theorem prover) you can automate this checking. At the moment writing Lean proofs takes ~10x longer than doing it by hand, but it gives you greates scalability through lowering the trust barrier. And presumably the cost won’t be 10x forever.

It also simplifies change management. Changing one parameter no longer means having to go through the whole proof by hand.

This pattern matches what large software engineering organizations have been doing for a while. Automated testing, types systems, …

Speaking of the 10x cost of formalizing proofs, this is something LLMs might be helpful for. Instead of formalizing everything by hand, you ask an LLM to do it for you. It’s not going to get it right every time, but at least it’s easy to verify whether it did or not. And even if it does not it might suggest an approach you can work out yourself.

Could software engineering learn from this? Is it finaly TLA+’s time to shine with the help of LLMs? Could other fields, physics, chemistry, have formal languages with automated checking?

I liked the Blueprint pattern a lot too: a human comes up with a “blueprint” of the proof that is linked to the Lean formalization, breaks down the work needed into smaller lemmas and definitions that can be tackled individually and provides a snapshot of progress.

Today I was playing with perf_event_open, a linux syscall that lets you set up all kinds of performance monitoring. One way to interact with the system is through the perf CLI tool. The problem?

root@X:~# cat print.py
print("Hello")

root@X:~# perf stat -e instructions:u python3 ./print.py
Hello

 Performance counter stats for 'python3 ./print.py':

          45713378      instructions:u

       0.030247842 seconds time elapsed

       0.025787000 seconds user
       0.004282000 seconds sys

45,713,378 is how many instructions it takes to initialize Python, print Hello and tear the whole thing down again. I want more precision! I want to only measure that single line of code.

Hence this little tool:

from cirron import Collector

with Collector() as c:
    print("Hello")

print(c.counters)
# Sample(instruction_count=17181, time_enabled_ns=92853)

Why would you want this? I was mostly just curious, but that’s not to say it can’t be useful: let’s say you really, really care about performance of your foo and you want to have a regression test that fails if it gets slower than a threshold. So you time.time() it, assert less than a threshold and… you realize your CI box is running a ton of concurrent tests and the timing is very noisy.

Not (as much) the instruction count!

This graph shows distribution of time measurements of the same piece of code using time.time() and time.perf_counter() and the instruction count measured with Cirron, scaled to fit with the time measurement². Note how impeccably tight the instruction count distribution is (also note though how it’s not completely constant).

This is not to say instruction count is the be-all and end-all. Equal instruction counts can have wildly different wall clock timings, etc. Still, it’s a useful tool if you’re aware of its limitations.

Here’s the Github repo, it’s very janky and only runs on Linux at the moment, thought I’ll probably add Apple Arm support soon. On the off-chance that there isn’t a tool out there that does this way better already and you’d find it useful if this was on PyPI or exposed more perf events or worked with $LANGUAGE, etc., let me know.

Backlinks

Discussions on /r/python and /r/programming subreddits.

First, instead of using hyperfine I’ve used Daniel Lemire’s (an absolute SIMD beast by the way, highly recommend reading his blog and marveling at simdjson) benchmarking harness and instead of a shared CPU I’ve used a dedicated host. I’ve also used C++ with -O1, which generates basically the same code as my handwritten assembly, but is much easier to work with.

As a refresher, we had two implementations of sum, one I expected to be slower

int result = 0;

for (int i = 0; i <= N/2; i++) {
    result += arr[2*i];
    result += arr[2*i+1];
}

return result;

And one I expected to be faster, thanks to the two operations in the loop being independent of each other, allowing out-of-order execution:

int result1 = 0;
int result2 = 0;

for (int i = 0; i <= N/2; i++) {
    result1 += arr[2*i];
    result2 += arr[2*i+1];
}

return result1 + result2;

With hyperfine on a shared CPU it was impossible to distinguish which is which. With perf events, I’m able to measure a consistent difference: the single variable implementation finishes in 293ms and the two variable in 261ms, ie. about 12% faster.

What’s more interesting though, is what happens when I unroll the loop even more and use 1, 2 or 4 variables:

int result1 = 0;
int result2 = 0;
int result3 = 0;
int result4 = 0;

for (int i = 0; i <= N/4; i++) {
    result1 += arr[2*i];
    result2 += arr[2*i+1];
    result3 += arr[2*i+2];
    result4 += arr[2*i+3];
}

return result1 + result2 + result3 + result4;

The one variable version gets slightly faster at 284ms, presumably due to lower loop overhead, but the 2 and 4 variable version speed up considerably more to 187ms and 170ms respectively, ie. 35% faster!

Unsurprisingly these differences disappear when I use -O3.

Pretty cool! Wish I had a more precise understanding of why this happen though.

Optimizing compilers tend to rewrite the human-obvious sum implementation to use multiple registers to keep track of parts of the total sum, adding them up at the end. For example here’s an implementation of sum in Rust 1.75 with -C opt-level=3 and the associated disassembly (shortened to only keep the relevant bits):

pub fn sum(arr: &[i32]) -> i32 {
    let mut result = 0;

    for n in arr.iter() {
        result += n
    }

    result
}

sum:
        ; Set xmm0 to 0.
        pxor    xmm0, xmm0
        xor     eax, eax
        ; Set xmm1 to 0.
        pxor    xmm1, xmm1 ; Set xmm1 to 0
.loop:
        ; Move 4 i32 values from our array into xmm2
        movdqu  xmm2, xmmword ptr [rdi + 4*rax]
        ; Add the 4 i32 values in xmm2 to xmm0.
        paddd   xmm0, xmm2
        ; Move the next 4 i32 values from our array into xmm2.
        movdqu  xmm2, xmmword ptr [rdi + 4*rax + 16]
        ; Add the 4 i32 values in xmm2 to xmm1, not xmm0 this time
        paddd   xmm1, xmm2

        ; Move index by 8 and if we're not done jump back to loop.
        add     rax, 8
        cmp     r8, rax
        jne     .loop

        ; If we are done, add up what's in xmm0 and xmm1 and return.
        paddd   xmm1, xmm0
        pshufd  xmm0, xmm1, 238
        paddd   xmm0, xmm1
        pshufd  xmm1, xmm0, 85
        paddd   xmm1, xmm0
        movd    eax, xmm1
        ret

As expected the compiler splits the sum between two registers (xmm0, xmm1). My hypothesis, therefore, was that the following hand-written assembly:

sum:
        lea     rcx, [rdi + 4*rsi]
        xor     eax, eax    
.loop:
        ; Use EAX for both additions
        add     eax, dword [rdi]    
        add     eax, dword [rdi + 4]
        lea     rdi, [rdi + 8] 
        cmp     rdi, rcx      
        jb      .loop        
        ret                         

Would be measurably slower than this multi-register one.

sum:
        lea     rcx, [rdi + 4*rsi]
        xor     eax, eax
        xor     ebx, ebx
.loop:
        ; Split the addition between EAX and EBX
        add     eax, dword [rdi]
        add     ebx, dword [rdi + 4]
        lea     rdi, [rdi + 8]
        cmp     rdi, rcx
        jb      .loop
        add     eax, ebx
        ret

Except… It’s not? If anything the first version is slightly faster.

Is my measuring methodology horrible (hyperfine on shared vCPU isn’t great, but I would expect it to be merely noisy, not biased)? Is this register renaming making things efficient behind the scenes?

Curious!