-
Notifications
You must be signed in to change notification settings - Fork 224
Rest API
The FACT Rest API intends to offer close to 100 % functionality of FACT in a script-able and integrate-able interface. The API does not comply with all REST guidelines perfectly, but aims to allow understandable and efficient interfacing.
All data send as message body and all data received from the API is of application/json
format. This allows easy integration into most programming languages, most commonly javascript and python. Since json does not support binary data, all binary data is additionally encoded in base64.
The API offers a number of URL parameters to go with GET requests. The parameters for each endpoint are listed via tables at the beginning of the endpoint descriptions below. They come with a usage example, but there will be a handful of usage examples listed after the introduction. Aside from the URL parameters the following descriptions provide information on each endpoint regarding:
- What it is for
- How to use it
- What the response looks like
All errors in processing a request to one of the listed endpoints result in an error message of the form:
{
"error_message": <string>, # Reason for the error
"request": { # List of (supported) parameters in the request
"limit": null,
"offset": null
},
"request_resource": "/rest/firmware", # Targeted endpoint
"status": 1, # 1 signifies error, 0 for all successful requests
"timestamp": 1587639454 # Unix timestamp of request
}
Successful messages will basically look the same, aside from status
being 0
and no error_message
key being present. In place of the error message successes will typically have an additional field containing requested information or processing feedback.
- Firmware
- FileObject
- Analysis
- Compare
- Binary
- Binary Search
- Statistics
- System Status
- Missing Analyses
- Authentication
Upload a firmware and query the firmware database.
Post without further parameters. The HTTP body must contain a json document of the following structure:
{
"device_name": <string>,
"device_part": <string>, # new in FACT 2.5
"device_class": <string>,
"file_name": <string>,
"version": <string>, # supersedes firmware_version field
"vendor": <string>,
"release_date": <string>,
"tags": <string>,
"requested_analysis_systems": <list>,
"binary": <string(base64)>
}
{
"uid": <string>, # new unique identifier for uploaded firmware
"status": 0, # 0 for success, 1 for an error
"request_resource": "/rest/firmware",
"timestamp": <integer>, # linux timestamp of request
"request": <json_document> # the input document
}
In case of an error, the uid will not be present, instead an error_message will be given. Also the status field will be 1.
Curl Sample:
curl http:https://localhost:5000/rest/firmware -X PUT -H "Content-Type: application/json" -d '{"vendor": "AVM", "device_class": "Router", "file_name": "rest_test.txt", "requested_analysis_systems": ["file_type", "file_hashes"], "binary": "dGVzdDEyMzQgdGhpcyBpcyBzb21lIHRlc3QgZmlsZQ==", "device_name": "rest_test", "firmware_version": "1", "release_date": "2011-01-01", "tags": "tag1,tag2"}'
Firmware analysis update
You can use this endpoint with PUT to update your firmware analysis. By using the following parameter, the listed analysis plugins are updated or added for all files in the database:
Parameter | Effect | Data format | Example |
---|---|---|---|
update | update firmware | json list containing plug in names | update=["cpu_architecture"] |
Curl sample:
curl -X PUT 'http:https://localhost:5000/rest/firmware/1754689233cfa0d65e27718529c0a94a7b2072775c6099fefc00eb41e70f6f71_73584640' -G --data-urlencode 'update=["software_components", "cpu_architecture"]'
Either request a single firmware by supplying the /[uid] parameter in the URL or browse the complete firmware database.
If a single firmware is requested, the result is of the form:
{
"firmware": <json_document>, # meta data and analysis results for the requested firmware
"status": 0, # 0 for success, 1 for an error
"request_resource": "/rest/firmware",
"timestamp": <integer>, # linux timestamp of request
"request": {
"uid": <string> # the requested uid
}
}
The default is to not aggregate summaries for the requested firmware. If you want to list summaries for the applied analysis plugins you have to add the summary parameter to the request.
Parameter | Effect | Data format | Example |
---|---|---|---|
summary | include summary in result | boolean [true, false] | summary=true |
Curl Sample: (Single Firmware)
curl 'http:https://localhost:5000/rest/firmware/e692eca8505b0f4a3572d4d42940c6d5706b8aabec6ad1914bd4d733be9dfecf_25221120' -X GET
To list all firmware in the database, or query a subset, access /rest/firmware
without a uid. To subset the database you can either page results using offset and limit or use a mongo syntax query for arbitrary subsets. A primer on the query syntax is embedded into the Web UI on the advanced search page.
Especially for analysis based subsets, matching based on included objects is more useful. To do so, apply the recursive flag. Using recursive you can get all firmware including at least one file matching the given query. This can further be combined with inverted. inverted and recursive combined result in a list of all firmware that does not include a single file matching the given query.
Here is the list of all applicable parameters:
Parameter | Effect | Data format | Example |
---|---|---|---|
offset | offset of results (paging) | integer (0 to ignore) | offset=10 |
limit | number of results (paging) | integer (0 to ignore) | limit=5 |
query | MongoDB style query (see advanced search) | json document | {"device_class": "Router"} |
recursive | Requires query. Query for parent firmware of matching objects | boolean [true, false] | recursive=true |
inverted | Requires query and recursive. Query for parent firmware that does not include the matching objects | boolean [true, false] | inverted=true |
Responses to firmware database queries look like this:
{
"uids": <list>, # list of uids for firmwares that match the request
"status": 0, # 0 for success, 1 for an error
"request_resource": "/rest/firmware",
"timestamp": <integer>, # linux timestamp of request
"request": {
"offset": <integer>, # 0 if not used
"limit": <integer>, # 0 if not used
"query": <json_document>, # empty if not used
"recursive": <boolean>, # false if not used
"inverted": <boolean> # false if not used
}
}
Curl Sample: (Multiple Firmware objects)
curl 'http:https://localhost:5000/rest/firmware?limit=3&offset=5' -X GET -G --data-urlencode 'query={"device_class": "Router"}'
Browse the file database or request specific file.
The /file_object endpoint works similar to the GET part of the /firmware endpoint. Instead of firmware this endpoint directly requests the included files. It allows to either request a specific file by providing the [uid] parameter in the URL or browse the database by applying query and/or limit and offset.
The only parameter applicable to the single file request is summary, which like for the firmware endpoint, includes summary lists for the applied plugins. This defaults to false
.
Parameter | Effect | Data format | Example |
---|---|---|---|
summary | include summary in result | boolean [true, false] | summary=true |
For possible responses see the /firmware endpoint. The only difference is that the firmware field in the single response is called file_object instead.
Curl Sample: (Single File)
curl 'http:https://localhost:5000/rest/file_object/6afa145520c15df916818997fd87202e8472141bb2ecf4869710cb026f15f9f8_28' -X GET
The usage of the browsing feature on files is also similar to /firmware. For explanation of the parameters see the documentation above. The supported parameters are:
Parameter | Effect | Data format | Example |
---|---|---|---|
offset | offset of results (paging) | integer (0 to ignore) | offset=10 |
limit | number of results (paging) | integer (0 to ignore) | limit=5 |
query | MongoDB style query (see advanced search) | json document | {"device_class": "Router"} |
Curl Sample: (Multiple Files)
curl 'http:https://localhost:5000/rest/file_object?limit=100' -G --data-urlencode 'query={"size": 512}' -X GET
Retrieve analysis results.
If you only want to retrieve the analysis result of a single analysis plugin for a specific file, you can do so by using this endpoint.
All you need to do is provide the UID
of the file and the name of the plugin.
Curl Sample:
curl -X GET 'http:https://localhost:5000/rest/analysis/366356382e9e2131cdcc8074fe7bc70d1960e899438dd1fc34cbf0d0549796f1_430168/file_type'
Responses will contain the result of the analysis plugin, e.g.
{
"analysis": {
"analysis_date": 1658148948.0259001,
"plugin_version": "1.0",
"result": {
"full": "ELF 32-bit MSB executable, MIPS, MIPS32 rel2 version 1 (SYSV), dynamically linked, interpreter /lib/ld-uClibc.so.0, stripped",
"mime": "application/x-executable"
},
"summary": [
"application/x-executable"
],
"system_version": null,
"tags": {}
},
"request": {
"plugin": "file_type",
"uid": "366356382e9e2131cdcc8074fe7bc70d1960e899438dd1fc34cbf0d0549796f1_430168"
},
"request_resource": "/rest/analysis",
"status": 0,
"timestamp": 1708331655
}
In case the analysis can't be retrieved (e.g. when the plugin did not run for this file), the response will contain a field "error_message"
.
Issue comparisons and retrieve comparison results.
Start the comparison of a given list of uids. The uid_list must contain uids of already analysed FileObjects or Firmware objects. The body then contains a json document of the following structure:
{
"uid_list": <list>,
"redo": <Boolean> # optional. if 'True' existing comparisons are overwritten
}
The response will contain a status field among others. If the status is 1 an error_message field is added:
{
"message": <string>, # optional. If status == 0, a notice is provided
"status": 0, # 0 for success, 1 for an error
"request_resource": "/rest/compare",
"timestamp": <integer>, # linux timestamp of request
"request": <json_document>, # see request format
"error_message": <string> # optional. Not present in for status == 0
}
A comparison result can be requested by GET request on the /rest/compare/<compare_id> endpoint, by providing a semicolon separated list of uids as compare_id (see example below). The response will contain a json_document with the comparison result, along with the fields status, timestamp, request_resource and request as meta data.
Curl Sample: (PUT)
curl http:https://localhost:5000/rest/compare -X PUT -H "Content-Type: application/json" -d '{"uid_list": ["2d63c2bcc2a9e1618c79310fdd2b77a83600a499f77c809a535e045aa499b169_42659840", ...], "redo": false}'
Curl Sample: (GET)
curl 'http:https://localhost:5000/rest/compare/6a4cba6f73a25f179e4d42460e9383ba499f93e12a9d6d81a7bdf0ea9d936a13_566112;f432dcff65513707659d2d6f3dbd43eefcc580b8c1a48cd3c1ef17a51d71cbb3_564500' -X GET
Request the binary of a given Firmware or FileObject.
Request a binary by providing the uid of the corresponding object in the URL. You can alternatively use the tar parameter on the request to get the target archive as its content repacked into a .tar.gz.
Parameter | Effect | Data format | Example |
---|---|---|---|
tar | Get tar.gz packed contents of target | boolean [true, false] | tar=true |
The response contains the binary or .tar.gz in base64 encoding as well as the SHA-256 sum of the binary as checksum and the original name of the file:
{
"SHA256": <string>, # SHA-256 checksum of raw binary (not of base64 encoding)
"binary": <string>, # base64 representation of the binary
"file_name": <string>, # original name of the downloaded file
"request":{
"uid": <string> # the requested uid
},
"request_resource": "/rest/binary",
"status": 0, # 0 for success, 1 for an error
"timestamp": <integer> # linux timestamp of request
}
Curl Sample:
curl http:https://localhost:5000/rest/binary/ceb0b13da1e765ec105460cf68367b78ed78b99fcbdd7654999a07b5b87e8f16_31 -X GET
Start a binary search on the binary database (or optionally on a single firmware) and fetch the results.
Start a binary search on the binary database by providing a set of YARA rules in the request data. The request data should have the form:
{
"rule_file": <string>, # a string containing the YARA rules
"uid": <string> # optional: the firmware UID
}
The UID is optional and only needs to be included if the search should be executed on the files of a single firmware.
The response data contains a search_id
which is used to fetch the search results.
The POST only initiates the search.
In order to get the search results, please use the GET interface.
curl 127.0.0.1:5000/rest/binary_search -X POST -d '{"rule_file": "rule rulename {strings: $a = \"OpenSSL\" condition: $a }", "uid": "715947e9e69aefcde3c09fe0b769a7210b0646b542bd8d958e3902eacb661ca7_2551176"}' -H "Content-Type: application/json"
Get the results of a previously initiated binary search.
The search_id
(provided by the POST request) is needed to fetch the corresponding search result.
The result of the search request can only be fetched once.
After this, the search needs to be started again.
The results have the form:
{
'binary_search_results': {
'<rule_name_1>': ['<matching_uid_1>', ...],
'<rule_name_2>': [...],
...
},
...
}
curl -X GET 127.0.0.1:5000/rest/binary_search/77479d57ad4c24cc471413147e1eb456915cde22ae07a3c2761d95d1eb5d9545_1533566955.8252065
Query some or all the statistics of FACT.
Retrieves all statistics or a particular statistic from the database as raw JSON data.
To list all the available statistics in the database access /rest/statistics
.
In case all statistics are requested, the result is of the form:
{
"architecture":{
"cpu_architecture": [[<stat_data>], ...]
},
"crypto_material":{
"crypto_material": [[<stat_data>], ...]
},
"elf_executable":{
"executable_stats": [[<stat_data>], ...]
},
"file_type":{
"file_types": [[<stat_data>], ...],
"firmware_container": [[<stat_data>], ...]
},
"firmware_meta":{
"device_class": [[<stat_data>], ...],
"vendor": [[<stat_data>], ...]
},
"general":{
"average_file_size": <float>,
"average_firmware_size": <float>,
"benchmark": <float>,
"creation_time": <float>,
"number_of_firmwares": <int>,
"number_of_unique_files": <int>,
"total_file_size": <int>,
"total_firmware_size": <int>
},
"ips_and_uris":{
"ips_v4": [[<stat_data>], ...],
"ips_v6": [[<stat_data>], ...],
"uris": [[<stat_data>], ...]
},
"known_vulnerabilities":{
"known_vulnerabilities": [[<stat_data>], ...]
},
"malware":{
"malware": [[<stat_data>], ...]
},
"release_date":{
"date_histogram_data": [[<stat_data>], ...]
},
"software_components":{
"software_components": [[<stat_data>], ...]
},
"unpacking":{
"average_packed_entropy": <float>,
"average_unpacked_entropy": <float>,
"data_loss_file_types": [[<stat_data>], ...],
"overall_data_loss_ratio": <float>,
"overall_unpack_ratio": <float>,
"packed_file_types": [[<stat_data>], ...],
"used_unpackers": [[<stat_data>], ...]
}
}
Curl Sample: (All Available Statistics)
curl -X GET 127.0.0.1:5000/rest/statistics
The optional parameter stat_name
can be used to retrieve only a specifically chosen statistic, as shown in the example below.
If a single statistic is requested, the result is of the form:
{
"architecture":{
"cpu_architecture": [[<stat_data>], ...]
}
}
Curl Sample: (Single Statistic)
curl -X GET 127.0.0.1:5000/rest/statistics/architecture
Request the system status of FACT.
Request a json document showing the system state of FACT, similar to the system health page of the GUI.
The results have the form:
{
"status": 0,
"timestamp": 1585810371
"request_resource": "/rest/status",
"plugins": {
"binwalk": {
"description": "binwalk signature and entropy analysis",
"version": "0.5.2"
},
[...]
},
"system_status": {
"backend": {
"_id": "backend",
"last_update": 1585810370.731242,
"name": "backend",
"platform": {
"fact_version": "3.1-dev",
"os": "Ubuntu 18.04",
"python": "3.6.9"
},
"status": "online",
"system": {
"cpu_cores": 8,
"disk_percent": 29.8,
"load_average": "0.4, 0.45, 0.33",
"memory_percent": 18.0,
[...]
},
"unpacking": {
"unpacking_queue": 0
}
"analysis": {
"analysis_main_scheduler": 0,
"plugins": {
"binwalk": {
"active": 0,
"queue": 0
},
[...]
}
},
},
"database": {
"_id": "database",
[...]
},
"frontend": {
"_id": "frontend",
[...]
}
},
}
Curl Sample:
curl http:https://localhost:5000/rest/status -X GET
Search the database for missing entries.
Search for:
- Missing files: Files whose UID is contained in the
files_included
of some object but that are not found in the database - Missing analyses: File objects that are lacking analyses which were performed on their parent firmware
The results have the form:
{
"missing_files": {
"<Parent FW/FO UID>": ["<missing file 1 UID>", ...],
...
},
"missing_analyses": {
"<Root FW UID>": ["<file 1 missing analysis UID>", ...],
...
},
"request_resource": "/rest/missing",
"status": 0, # 0 for success, 1 for an error
"timestamp": <integer> # linux timestamp of request
}
Curl Sample:
curl http:https://localhost:5000/rest/missing -X GET
If the authentication option is enabled in the configuration, each REST request has to be accompanied by an API key in the HTTP header. For more information regarding the authentication in general and which function needs what level of privilege see FACT authentication.
If your user has been given the exemplary key ABCDEFG=
a request to the /rest/binary
endpoint in accordance to the example given above would look like this:
curl http:https://localhost:5000/rest/binary/ceb0b13da1e765ec105460cf68367b78ed78b99fcbdd7654999a07b5b87e8f16_31 -X GET --header 'Authorization: ABCDEFG=' -L