Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stack traces to trace.py #712

Merged
merged 1 commit into from
Sep 30, 2016
Merged

Add stack traces to trace.py #712

merged 1 commit into from
Sep 30, 2016

Conversation

palmtenor
Copy link
Member

As mentioned in #621 , this would be very useful for debugging. This Diffs adds -U and -K options to trace.py so that it would print out user / kernel stack traces on each event.

Sample generated BPF program and outpout:

qinteng@devbig473:~ {}$ sudo LD_LIBRARY_PATH=/data/users/qinteng/toolchains/bcc-dev/lib64: /data/users/qinteng/toolchains/python-2.7.11/bin/python /data/users/qinteng/bcc/tools/trace.py finish_task_switch -K -U -M 3 -v

#include <linux/ptrace.h>
#include <linux/sched.h>        /* For TASK_COMM_LEN */


struct probe_finish_task_switch_1_data_t
{
        u64 timestamp_ns;
        u32 pid;
        char comm[TASK_COMM_LEN];

       int kernel_stack_id;
       int user_stack_id;
};

BPF_PERF_OUTPUT(probe_finish_task_switch_1_events);
BPF_STACK_TRACE(probe_finish_task_switch_1_stacks, 1024);


int probe_finish_task_switch_1(struct pt_regs *ctx)
{

        u32 __pid = bpf_get_current_pid_tgid();
        if (__pid == 911113) { return 0; }



        if (!(1)) return 0;

        struct probe_finish_task_switch_1_data_t __data = {0};
        __data.timestamp_ns = bpf_ktime_get_ns();
        __data.pid = bpf_get_current_pid_tgid();
        bpf_get_current_comm(&__data.comm, sizeof(__data.comm));


        __data.user_stack_id = probe_finish_task_switch_1_stacks.get_stackid(
          ctx, BPF_F_REUSE_STACKID | BPF_F_USER_STACK
        );
        __data.kernel_stack_id = probe_finish_task_switch_1_stacks.get_stackid(
          ctx, BPF_F_REUSE_STACKID
        );
        probe_finish_task_switch_1_events.perf_submit(ctx, &__data, sizeof(__data));
        return 0;
}

p::finish_task_switch FLT=1 ACT=[]/[]
TIME     PID    COMM         FUNC
21:19:20 7565   TierUpdNotif finish_task_switch
    User Stack Trace:
        00007fbf96ec0a83 [unknown]
        000000000247d071 event_base_loop
        0000000002190046 folly::EventBase::loopBody(int)
        0000000000f9df02 facebook::services::TierUpdatesObserver::notifierThread()
        00007fbf96b44ec1 execute_native_thread_routine
        00007fbf978af7f1 start_thread
        00007fbf96ec046d __clone
    Kernel Stack Trace:
        ffffffff8108e0f1 finish_task_switch
        ffffffff81774cf7 schedule
        ffffffff817777dc schedule_hrtimeout_range_clock
        ffffffff81777823 schedule_hrtimeout_range
        ffffffff811fd3fc ep_poll
        ffffffff811fe7f5 sys_epoll_wait
        ffffffff81778232 system_call_fastpath

21:19:20 0      swapper/11   finish_task_switch
    User Stack Trace:
        -14
    Kernel Stack Trace:
        ffffffff8108e0f1 finish_task_switch
        ffffffff81774cf7 schedule
        ffffffff81774f9e schedule_preempt_disabled
        ffffffff810a6ca8 cpu_startup_entry
        ffffffff81031760 start_secondary

21:19:20 862    fct0-smp     finish_task_switch
    User Stack Trace:
        -14
    Kernel Stack Trace:
        ffffffff8108e0f1 finish_task_switch
        ffffffff81774cf7 schedule
        ffffffff817773a7 schedule_timeout
        ffffffff810c0460 msleep_interruptible
        ffffffffa000328e kfio_msleep_noload
        ffffffffa009f635 ifio_8a30f.3f931e2037196fc46b5b711d5ae377442c8.3.2.4.1086

Copy link
Collaborator

@goldshtn goldshtn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Great work, this will be very useful!

@4ast 4ast merged commit 6b0ed37 into iovisor:master Sep 30, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants