Skip to content

Commit

Permalink
tools/inject: Update documentation
Browse files Browse the repository at this point in the history
  • Loading branch information
gobenji authored and yonghong-song committed Jul 12, 2019
1 parent 86633c4 commit 5933839
Show file tree
Hide file tree
Showing 2 changed files with 93 additions and 13 deletions.
83 changes: 77 additions & 6 deletions man/man8/inject.8
Original file line number Diff line number Diff line change
@@ -1,21 +1,27 @@
.TH inject 8 "2018-03-16" "USER COMMANDS"


.SH NAME
inject \- injects appropriate error into function if input call chain and
predicates are satisfied. Uses Linux eBPF/bcc.


.SH SYNOPSIS
.B inject -h [-I header] [-P probability] [-v] [-C count] mode spec
.B inject -h [-I header] [-P probability] [-v] [-c count] <mode> <spec>


.SH DESCRIPTION
inject injects errors into specified kernel functionality when a given call
chain and associated predicates are satsified.
chain and associated predicates are satisfied.

WARNING: This tool injects failures into key kernel functions and may crash the
kernel. You should know what you're doing if you're using this tool.

This makes use of a Linux 4.16 feature (bpf_override_return())

Since this uses BPF, only the root user can use this tool.
.SH REQUIREMENTS
CONFIG_BPF, CONFIG_BPF_KPROBE_OVERRIDE, bcc


.SH OPTIONS
.TP
\-h
Expand All @@ -30,20 +36,85 @@ Necessary headers to be included.
\-P probability
Optional probability of failure, default 1.
.TP
\-C count
\-c count
Number of errors to inject before stopping, default never stops.


.SH MODE

.TP
\fBkmalloc\fR
Make the following function indicate failure
.RS 14
int should_failslab(struct kmem_cache *s, gfp_t gfpflags)
.RE

.TP
\fBbio\fR
Make the following function indicate failure
.RS 14
int should_fail_bio(struct bio *bio)
.RE

.TP
\fBalloc_page\fR
Make the following function indicate failure
.RS 14
bool should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)
.RE


.SH SPEC
.B FUNCTION([ARGS])[(TEST)] [=> ...]

A list of predicates separated by "=>". A predicate is a function signature
(name and arguments) in a call stack and a test on the function's arguments.

Missing predicates are implicitly true. Missing tests are implicitly true.
Specifying the function arguments is optional if the test does not use them.
If the error injection function is not listed as the first predicate, it is
implicitly added.

Functions are listed in the reverse order that they are called, ie. if a()
calls b(), the spec would be "b() => a()".


.SH REQUIREMENTS
CONFIG_BPF, CONFIG_BPF_KPROBE_OVERRIDE, bcc


.SH EXAMPLES
Please see inject_example.txt
.EX
inject kmalloc -v 'SyS_mount()'
.EE

.EX
inject kmalloc -v 'mount_subtree() => btrfs_mount()'
.EE

.EX
inject -P 0.5 -c 100 alloc_page "should_fail_alloc_page(gfp_t gfp_mask, unsigned int order) (order == 1) => qlge_refill_bq()"
.EE

Please see the output of '-h' and tools/inject_example.txt for more examples.


.SH SOURCE
This is from bcc.
.IP
https://github.com/iovisor/bcc
.PP
Also look in the bcc distribution for a companion _examples.txt file containing
example usage, output, and commentary for this tool.


.SH OS
Linux


.SH STABILITY
Unstable - in development.


.SH AUTHOR
Howard McLauchlan
23 changes: 16 additions & 7 deletions tools/inject_example.txt
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,16 @@ The "(true)" without an associated function is a predicate for the error
injection mechanism of the current mode. In the case of kmalloc, the predicate
would have access to the arguments of:

int should_failslab(struct kmem_cache *s, gfp_t gfpflags);
should_failslab(struct kmem_cache *s, gfp_t gfpflags)

The bio mode works similarly, with access to the arguments of:

static noinline int should_fail_bio(struct bio *bio)
Other modes work similarly.
"bio" has access to the arguments of:

should_fail_bio(struct bio *bio)

"alloc_page" has access to the arguments of:

should_fail_alloc_page(gfp_t gfp_mask, unsigned int order)

We also note that it's unnecessary to state the arguments of the function if you
have no intention to reference them in the associated predicate.
Expand Down Expand Up @@ -62,7 +67,7 @@ In general, it's worth noting that the required specificity of the call chain is
dependent on how much granularity you need. The example above might have
performed as expected without the intermediate btrfs_alloc_device, but might
have also done something unexpected(an earlier kmalloc could have failed before
the one we were targetting).
the one we were targeting).

For hot paths, the approach outlined above isn't enough. If a path is traversed
very often, we can distinguish distinct calls with function arguments. Let's say
Expand Down Expand Up @@ -115,12 +120,14 @@ fail our mounts half the time:
# ./inject.py kmalloc -v -P 0.01 'SyS_mount()'

USAGE message:
usage: inject.py [-h] [-I header] [-P probability] [-v] {kmalloc,bio} spec
usage: inject.py [-h] [-I header] [-P probability] [-v] [-c COUNT]
{kmalloc,bio,alloc_page} spec

Fail specified kernel functionality when call chain and predicates are met

positional arguments:
{kmalloc,bio} indicate which base kernel function to fail
{kmalloc,bio,alloc_page}
indicate which base kernel function to fail
spec specify call chain

optional arguments:
Expand All @@ -130,6 +137,8 @@ optional arguments:
-P probability, --probability probability
probability that this call chain will fail
-v, --verbose print BPF program
-c COUNT, --count COUNT
Number of fails before bypassing the override

EXAMPLES:
# ./inject.py kmalloc -v 'SyS_mount()'
Expand Down

0 comments on commit 5933839

Please sign in to comment.