A tool for visualizing the relationship between generative fuzzing inputs and code coverage.
JQF generators operate by consuming a stream of pseudorandom bytes, which are then used to produce the values returned
by Random
methods like nextInt()
and nextChar()
.
The eye-fuzz
frontend shows the program locations (stack traces) at which these random bytes are consumed, and
allows you to modify those values to influence the generator output to your liking.
Compilation requires the following build tools:
This project is configured to build and run under JDK/JRE 8 - it may work on newer versions of Java, but it has not been tested.
After cloning this repository, run the following two commands:
mvn package # Builds the server backend
cd client && yarn # Builds the frontend UI
Once you've done so, you're ready to try out your generator!
First, write your generator and test driver in a separate directory (see the Zest wiki for an example of how to do this).
Next, start up the server with the following command:
./server.sh [-c <classpath>] <args>
where <classpath>
is the build directory containing your JAR or class files of the test driver and generator.
This will start a local HTTP server on port 8000 that will track the state of your fuzzing session.
<args>
are as follows, in order:
GENERATOR_CLASS_NAME
: the name of the generator classTEST_CLASS_NAME
: the name of the test driver classTEST_METHOD_NAME
: the name of the method on the test driver to be invokedCOVERAGE_CLASS_NAMES
: comma-separate list of class names for Jacoco to collect coverage on (coverage is automatically collected on the test case class; pass empty quotes to specify no additional classes)
For example, suppose you compiled JQF's calendar tutorial
in the folder ../calendar-example
. To start the fuzzing server and collect coverage for the CalendarLogic
and
CalendarGenerator
classes, you would run the following command:
./server.sh -c ../calendar-example/ CalendarGenerator CalendarTest testLeapYear "CalendarLogic,CalendarGenerator"
Lastly, open client/index.html
to begin fuzzing.
Once the server is running, you can see the values returned by Random
method calls in the generator. To modify these
values and rerun the generator, fill in values corresponding to the desired function call(s) under the "New Value"
column, and press "Rerun generator" in the top left. The generator output should be updated accordingly.
To view test coverage for the current run, move to the "Coverage Data" tab. This displays a code coverage summary
produced by Jacoco. It also indicates whether the generator produced a valid
input to the test method in the "Test Status" field - an input that fails test preconditions or constraints will show
INVALID
, whereas a valid input shows SUCCESS
, FAILURE
, or TIMEOUT
depending on the result of the test run.
To load an input (whether previously saved by eye-fuzz or produced by a Zest fuzzing session), drag the file into the
saved_inputs
directory. After refreshing the page, select the file name from the "Load input file" dropdown and hit
Load
.
-
Rerun generator: Reruns the generator with the new values specified.
-
Save last input to file: Saves the sequence of bytes that produced the last generator result to a file. This file is just a binary sequence that can be fed to any generator, and can be used with Zest's ReproGuidance.
The file is saved in the
saved_inputs
directory. -
Load input file: Loads a sequence of bytes to be used for a generator from a file in the
saved_inputs
directory. To load a file from elsewhere in your filesystem, move or copy it into thesaved_inputs
folder and refresh the page.
- Rerun test case: Reruns the test case using the most recent generator output, updating the collected coverage data.
These options control how program locations and their corresponding values are displayed and formatted.
-
Show type-level decisions: When checked, instead of displaying a single entry for each individual byte, a single row is displayed for each call to a
Random
method.For example, suppose a call to
randInt()
produces the value0xABCD1234
. When this option is checked,0xABCD1234
is displayed; otherwise it shows four entries containing the values0x34
,0x12
,0xCD
, and0xAB
(from top to bottom).Note also that if a
Random
call is made with min or max bounds, type-level information will display the value returned to the generator rather than simply combining the bytes together. If a callrandInt(3, 7)
were to produce the raw bytes0xABCD1234
, the actual displayed value would be3
, as the bytes are clamped into the range [3, 7). -
Show unused locations: When checked, stack traces that weren't reached on this run (but were reached on previous ones) are shown.
-
Filter stack trace by class name: Provides a string to filter lines from the displayed stack traces. When not empty, only lines in stack traces that contain the specified string (case-sensitive) are shown.
-
Display format: Chooses how to format numeric values (decimal, binary, hexadecimal, or UTF-16 codepoint).
-
View history: Hitting the left arrow causes the display to show values and generator output from progressively older runs, while hitting the right arrow shows progressively newer ones.
These options let you save, load, and reset an eye-fuzz
fuzzing session, which captures the full history of changes
in values and generator outputs made by the user.
-
Restart from scratch: Hitting this button completely resets the state of your fuzzing session. The generator will be reseeded with a fresh set of bytes.
-
Save current session to file: Saves the whole history of choices made for the duration of a fuzzing session to a JSON file with the specified name (the
.json
extension is not automatically applied).The file is saved in the
saved_sessions
directory. -
Load session from file: Loads a history of choices from a JSON file, as produced by the "Save current session" command. This replaces the current fuzzing session, and the generator will be rerun once to ensure consistency.
eye-fuzz
may encounter difficulties in the following conditions.
eye-fuzz
supports all methods implemented by JUnit Quickcheck's SourceOfRandomness
API (found
here,
except the following:
nextBigInteger
nextInstant
nextDuration
Additionally, the following methods can be manipulated at the byte level by the frontend, but are not displayed properly with type or bounds information:
nextBytes
nextFloat
nextDouble
nextGaussian
If your use case requires support for any of these methods, please file an issue.
Because it is impossible to distinguish between function calls with identical stack traces, eye-fuzz
may
encounter unexpected behavior in tracking history when faced with such a scenario (behavior for modifying and generating
new bytes should be unaffected). Identical stack traces may occur if two calls to the same Random
method are placed on
the same line (for example, f(randInt(), randInt()))
), or if a Random
method call occurs in the body of a for loop.
If a generator caches intermediate computations that invoke Random
methods, then rerunning the generator twice with
the same underlying byte stream may produce different results (this is a limitation of JQF as well). More broadly, any
generator whose output is not solely a function of the input pseudorandom byte stream will exhibit this behavior.
Currently, the generator output displayed on the frontend is simply the result of calling toString()
on the produced
object (truncated to 1024 characters). Custom serialization functions are supported but not yet exposed to the
command-line interface - please file an issue if this is a feature you need.