Flowslicer extracts data flow slices from binaries. It can then ingest them into a database, then compare them with match set analysis.

See the related project REveal

Flowslicer requires Python 3.10 or newer.

A Binary Ninja license with "GUI-less processing" is needed to run the command line. There are plans to make it available purely through the plug-in interface to allow Binary Ninja users with personal licences to use this tool, but that is not yet implemented.

To install the Binary Ninja API, run scripts/install_api.py with your specific python interpreter.

Also install the cbor2 library using pip:

python -m pip install cbor2

usage: flowslicer.py [-h] [--db [PATH]] [--slices [PATH]] [--function NAME [NAME ...]] [--force-update] [--parallelism N] [-v] {search,ingest,slice} [files ...]

Data flow slicing and match set analysis tool.

positional arguments:
  {search,ingest,slice}
                        Command to run
  files                 List of binaries to slice, slice files to ingest, or slice files to search. Folders will be recursively enumerated.

options:
  -h, --help            show this help message and exit
  --db [PATH]           Database file for ingest and search. Defaults to "slices.slicedb".
  -v, --verbose

Slice command options:
  --slices [PATH]       Folder to place slice files and logs. Defaults to "data_flow_slices".
  --function NAME [NAME ...]
  --force-update
  --parallelism N

The slice command will take a list of binaries (or folders to enumerate binaries) and produce a .slices file for each binary. These .slices files will be placed in a folder specified by --slices (defaults to data_flow_slices).

These .slices files can then be processed and combined into a single database file using ingest command or used for search files using the search command.

Example usage:

./flowslicer.py slice sample_binaries/linux_bin/ --slices slices/linux_bin --parallelism 10

This command will process all executables in the sample_binaries\linux_bin folder, create a folder called slices/linux_bin/, and generate one .slices file for each binary in that folder. 900 linux binaries takes up to 90 minutes.

The data flow slicing process can take a long time, depending on the binaries. For example, on an Intel 12700H processor (14C 20T) --parallelism 10 will spin up 10 instances to load the cores to nearly 100%, and can process 900 linux files from /usr/bin in about 90 minutes.

Note that while each parallel instance will create a separate Binary Ninja (headless) instance, and each Binary Ninja instance is multi-threaded. However, some of the processing will be single-threaded in the python script, so a parallelism limit somewhere in the 50-75% of the physical cores is recommended to saturate processing resources.

The memory usage per instance can vary greatly per binary. With basic Linux and Windows system binaries and Binary Ninja 3.3, each binary probably uses an average of 1GB, with few binaries exceeding 2GB of memory. So a parallelism-to-memory ratio of 1:2GB should be sufficient for processing the majority of binaries.

However, some binaries can consume much more memory. For example, according to the Binary Ninja blog Linux Chrome with full symbols can consume ~34GB of memory. Since Flowslicer will output to a .temp file and rename to .slices only after processing completes, binaries that failed with out-of-memory can be retried with more memory (e.g. lower parallelism to increase memory ratio).

The ingest command takes a list of files with the .slices extension and combines them into a single .slicedb file.

./flowslicer.py ingest --db linux_bin.slicedb slices/linux_bin

Example output:

...
   691 slices,    428 unique slices\linux_bin\zdump.slices
  6929 slices,   2842 unique slices\linux_bin\zenity.slices
  9246 slices,   4309 unique slices\linux_bin\zipinfo.slices
  2095 slices,   1062 unique slices\linux_bin\[.slices
Merged counts 3309730
Wrote 2003995 items from 900 files to .\linux_bin.slicedb

The search command searches for data flow slices

./flowslicer.py search --db linux_bin.slicedb slices/linux_bin/git.slices

This produces match set output:

...
    49 47ca9d8276ce   3 emacs-gtk git-shell git
    49 3931ec1054bc   2 git snap
    55 bcab5267e51a   5 cmake cpack ctest git-shell git
    87 9f7c6da1158a   4 git-shell git qemu-system-i386 qemu-system-x86_64
   130 e33e41aaaf48   4 git-shell git vim.basic vim.tiny
   133 91a13c5c60a9   2 gdb git
   181 e3d779ca2571   3 gdb git-shell git
   229 b398592e85f6   3 bat git-shell git
 18357 3918f20021dc   2 git-shell git
 38031 e165421110ba   1 git

The first column is the number of slices in the match set. The second column is a hash of the set of files in this match set. This match set hash can be used to reference the slices in the folder output by the --detail OUT_DIR option (which defaults to match_set_detail/). The third column is the number of files, and the last is the list of files by name. Using this output, we can for example conclude there are 18357 slices that match git-shell and git, but no other files in the input database.

When running a search command with --detail OUT_DIR (which defaults to match_set_detail/), a text file is generated for each match set listing detailed information on all the slices that have that match set. For example:

Match set 3918f20021dc has 2 files
  git-shell
  git

Slices with the match set:
Slice 0001810b8c75 2 2 2      sub_1879c0           1879c9,1879fc
Slice 00022eea4ee9 2 2 2      sub_14f8c0           14f8de,14f91f,14fa6d,14fa8a,14fa9e
Slice 0002338e0e1a 2 2 2      sub_1b20a0           1b20d2,1b20d7,1b20de,1b20e9
Slice 0003bb5e377e 2 2 2      sub_1c0840           1c09d4,1c0a87,1c0a9e,1c0aac,1c0ac0,1c0e83,1c0f59,1c0fa7
...

This contains the following details:

• Slice hash
• Counts of the slice in other files (# files, # functions, # instances)
• From the search file, the function names and instruction addresses.

Further down the detail file, each slice is repeated but with the canonicalized text of the slice.

• Support not-of-interest databases that subtract out match sets that include known/common libraries.
• Run commands (slice, ingest, search) from within Binary Ninja.