- Docker (https://docs.docker.com/get-docker/)
Clone repo:
$ git clone https://github.com/cmusv/tenet.git
$ cd tenet
$ pip install -r requirements.txt
$ pip install .
$ ./setup.sh
usage: tenet [-h] [-d] [-q] [-v] [-vb] {run} ...
Fine-grained approach to detect and patch vulnerabilities
optional arguments:
-h, --help show this help message and exit
-d, --debug full application debug mode
-q, --quiet suppress all console output
-v, --version show program's version number and exit
-vb, --verbose Verbose output.
sub-commands:
{plugin,run}
plugin plugin controller
run Runs a workflow
usage: tenet plugin [-h] {install,uninstall} ...
optional arguments:
-h, --help show this help message and exit
sub-commands:
{enabled, install,uninstall}
enabled Lists enabled plugins
install Installs plugin
uninstall Uninstalls plugin
This command copies the target plugin file to the plugin_dir
location specified in the tenet.yml
config file
(default is ~/.tenet/plugins
) and enables the plugin in the config file.
Note:
- Some names are reserved for core handlers — e.g. workflow, container, command, etc.
- The name of the plugin must match the label attributed to the plugin.
usage: tenet plugin install [-h] -p PATH [-f] -n NAME
optional arguments:
-h, --help show this help message and exit
-p PATH, --path PATH File path of the plugin.
-f, --force Overwrites existing plugins.
-n NAME, --name NAME Name of the plugin (should match its label).
tenet plugin install -p ~/workdir/plugins/code2vec.py -n code2vec -f
usage: tenet plugin uninstall [-h] -n NAME
optional arguments:
-h, --help show this help message and exit
-n NAME, --name NAME Name of the plugin.
tenet plugin uninstall -n code2vec
usage: tenet run [-h] -f FILE -d DATASET -wd WORKDIR -b BIND
optional arguments:
-h, --help show this help message and exit
-f FILE, --file FILE Path to the pipeline config file
-d DATASET, --dataset DATASET
Path to the input csv dataset file
-wd WORKDIR, --workdir WORKDIR
Path to the workdir.
-b BIND, --bind BIND Docker directory path to bind (to workdir as a volume).
tenet -vb run -f ~/projects/code2vec/code2vec_pipe.yml -d ~/projects/code2vec/cwe79.tsv -wd ~/projects/code2vec -b /projects/code2vec
The schema consists of three components: nodes
, layers
, and workflow
.
A node can be a plugin or a container, and these that are defined in the nodes
, instantiated in the layers
and
executed in the workflow
.
This type of node is a Python plugin that extends the framework with a specific functionality.
- name: str
- name of the label of the plugin;
- kwargs: dict
- a variable number keyword arguments (keys and values) that are passed to the plugin;
- plugin:
name: preprocess
kwargs:
max_contexts: 800
- plugin:
name: code2vec
kwargs:
train: true
max_contexts: 800
This type of node executes commands in a specific container. The container shares a volume with its
respective node directory inside the working directory. Given the local working directory /home/user/projects/code2vec
,
the node directory for a container with the name jscodeshift
would be /home/user/projects/code2vec/jscodeshift
.
The binding with the working directory in the container is specified with the -b
flag. For instance,
the working directory in the container /projects/code2vec
corresponds
to the local working directory /home/user/projects/code2vec
.
- name: str
- name of the container
- image: str
- name of the container's image (must exist)
- cmds: List[str]
- list of commands to be executed in the container
- these are converted to ContainerCommands
- can contain placeholders
- output: str
- specifies the file name of the output, relative to the working directory of the container
- can contain placeholders
The following node instantiates a container with the name astminer
from the image astminer:latest
,
and executes the respective commands of astminer with different arguments provided from other nodes.
- container:
image: astminer:latest
name: astminer
cmds:
- "mkdir -p {p2}"
- "export NODE_OPTIONS=\"--max-old-space-size=8192\""
- "java -jar -Xms4g -Xmx4g build/shadow/astminer.jar code2vec {p1} {p2} {p3} 0"
output: "{p2}/path_contexts.c2s"
Consists of a sequential list of nodes
to be executed in the specified order (from top to bottom).
layers:
dataset:
- find_refs:
node: find_references
...
- raw:
node: github_collector
...
diff:
- labels:
node: labeler
...
- jscodeshift:
node: jscodeshift
...
.
.
.
For instance: find_refs
-> raw
-> labels
-> jscodeshift
-> ...
Consists of a sequential list of layers
to be executed in the specified order (from left to right).
workflow: [dataset, diff, prepare, evaluate]
For instance dataset
-> diff
-> prepare
-> evaluate
This project includes a number of helpers in the Makefile
to streamline common development tasks.
The following demonstrates setting up and working with a development environment:
### create a virtualenv for development
$ make virtualenv
$ source env/bin/activate
### run tenet cli application
$ tenet --help
### run pytest / coverage
$ make test
Included is a basic Dockerfile
for building and distributing Tenet
,
and can be built with the included make
helper:
$ make docker
$ docker run -it tenet --help