Skip to content

Commit

Permalink
Fix userspace stack unwinding on powerpc
Browse files Browse the repository at this point in the history
Nysal reported that the bcc tool offcputime.py does not display
userspace stack traces on powerpc. Looking at the bpf program maps
showed that stack traces were indeed being captured by the kernel, but
were not being displayed by bcc userspace. This turned out to be the
case since we were having the second entry in the stack trace as zero,
and bcc's stack walker correctly assumes that the stack trace ends when
a zero entry is encountered.

However, on powerpc, a perf callchain includes two additional entries
after the first so as not to miss any data that may be required for
stack unwinding. The first entry is always the nip (next instruction
pointer) which is always valid. The second entry is LR, the link
register, and the third entry is the value in the LR save area in the
(second) stack frame. Due to how stack frames are setup, LR or the entry
in the stack frame may be zero'ed out. In addition, with support for
system call vectored in the kernel, we are setting LR to zero due to how
the syscall interface now works. To disambiguate this, access to
debuginfo would be necessary to understand which entry is the correct
one to use. Since that isn't always possible, simply allow the stack
unwinding to proceed when encountering zero in the second or third
entry.

Before this patch:
$ sudo ./offcputime.py -uU
Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.

^C
    write
    -                python (7784)
        8

    write
    -                sudo (7782)
        12

    [unknown]
    -                multipathd (685)
        5002377

    clock_nanosleep
    -                multipathd (697)
        5002485

    __poll
    -                python (7784)
        5122835

    ppoll
    -                sshd (7754)
        5147863

    ppoll
    -                sudo (7782)
        5148540

After this patch:
$ sudo ./offcputime.py -uU
Tracing off-CPU time (us) of user threads by user stack... Hit Ctrl-C to end.

^C
    write
    [unknown]
    [unknown]
    sudo_ev_loop_v1
    sudo_ev_dispatch_v1
    [unknown]
    [unknown]
    [unknown]
    [unknown]
    __libc_start_main
    -                sudo (7827)
        12

    clock_nanosleep
    [unknown]
    nanosleep
    sleep
    [unknown]
    [unknown]
    __clone
    -                multipathd (697)
        1000518

    [unknown]
    [unknown]
    pthread_cond_wait
    [unknown]
    [unknown]
    [unknown]
    __libc_start_main
    -                multipathd (685)
        1000548

    __poll
    [unknown]
    perf_reader_poll
    [unknown]
    [unknown]
    [unknown]
    [unknown]
    _PyObject_MakeTpCall
    PyObject_Vectorcall
    _PyEval_EvalFrameDefault
    PyEval_EvalCode
    [unknown]
    [unknown]
    [unknown]
    _PyRun_SimpleFileObject
    _PyRun_AnyFileObject
    Py_RunMain
    [unknown]
    Py_BytesMain
    [unknown]
    __libc_start_main
    -                python (7829)
        2069828
  • Loading branch information
rnav authored and yonghong-song committed Jan 29, 2024
1 parent 1a784cf commit 171ef76
Showing 1 changed file with 4 additions and 2 deletions.
6 changes: 4 additions & 2 deletions src/python/bcc/table.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
import errno
import re
import sys
import platform

from .libbcc import lib, _RAW_CB_TYPE, _LOST_CB_TYPE, _RINGBUF_CB_TYPE, bcc_perf_buffer_opts
from .utils import get_online_cpus
Expand Down Expand Up @@ -1193,8 +1194,9 @@ def next(self):
else:
addr = self.stack.ip[self.n]

if addr == 0 :
raise StopIteration()
# powerpc populates optional, potentially-null second and third entries
if addr == 0 and (not platform.machine().startswith('ppc') or (self.n != 1 and self.n != 2)) :
raise StopIteration()

return self.resolve(addr) if self.resolve else addr

Expand Down

0 comments on commit 171ef76

Please sign in to comment.