A potential issue with this approach is that each sample buffer consumes at least one page of locked memory and some users might have RLIMIT_MEMLOCK limits that rr could run into. However the distros I've looked at so far either default to unlimited mlocked memory, or have a low soft limit but a very high hard limit (1TB on Fedora).

What would that result in, performance-wise? Would that be the single implementation or would there be an initial check (which we already have) and an option to switch to this handling?

I'm planning to have it always enabled. That simplifies things.

Unfortunately it looks like this doesn't work because the kernel does not generate SIGIO for ring-buffer watermark overflows... it marks the event fd readable (generating poll events), but not the signal.

Arguably this is a kernel bug. The man page says

Overflow conditions can be  captured  by
       monitoring the event file descriptor with poll(2), select(2), or epoll(7).  Alternatively, the overflow events can be cap?
       tured via sa signal handler, by enabling I/O signaling on the file descriptor
...
       There are two ways to generate overflow notifications.

       The  first  is  to set a wakeup_events or wakeup_watermark value that will trigger if a certain number of samples or bytes
       have been written to the mmap ring buffer.  In this case, POLL_IN is indicated.

Testcase here: https://gist.github.com/rocallahan/a046f28800383fbe2e2904aa8507fcfa

It looks like perf sample events trigger SIGIO here: https://github.com/torvalds/linux/blob/976600c6da56c488776c9ee2a5733ae9736e2a75/kernel/events/core.c#L6685

But watermark wakeups go through perf_output_wakeup here: https://github.com/torvalds/linux/blob/976600c6da56c488776c9ee2a5733ae9736e2a75/kernel/events/ring_buffer.c#L20

It's sort of maddening that "trigger FASYNC signals" and "notify poll waiters" require separate code instead of going through a utility function that does both.

With that kernel bug fixed, the following branch pretty much works:
https://github.com/rr-debugger/rr/tree/watermark
The changes are simple.

If/when that kernel fix lands upstream and becomes available, I'll land the rr side. TODO:

• Dynamically configure rr to use PERF_RECORD_SWITCH if the kernel supports it
• Investigate using PERF_EVENT_IOC_SET_OUTPUT to share mmap buffers