initial results of the mal2 project

README.md

@@ -1,2 +1,137 @@
-# mal2-fakeshop-detection_detector-api
-MAL2 Fake-Shop Detector API integration of known legit and fraudulent sites, integration of detector-models, prediction, API for serving plugin interaction 
+# Fake-Shop Detector API
+
+## About / Synopsis
+
+* central source for serving browser plugin
+* homogeneously integrates multiple local or api providers of trustworthy and known fraudulent sites
+* provides mal2-model prediction capabilities
+* caching and load balancing
+* Project status: working/prototype
+
+## Table of contents
+
+> * [Requirements](#requirements)
+> * [Installation](#installation)
+> * [Usage](#usage)
+> * [Running with Docker](#Running-with-Docker)
+> * [REST-API](#rest-api)
+> * [About MAL2](#about-mal2)
+> * [Contact](#contact)
+> * [License](#license)
+
+## Requirements
+
+* Ubuntu 16.04
+* Python 3.8
+* PostgreSQL 10
+* Python-Packages as defined in [requirments.txt](backend-api-server/requirements.txt)
+
+## Installation
+
+Create a Python virtual environment with e.g. [virtualenvwrapper](https://virtualenvwrapper.readthedocs.io/en/latest/) or anaconda.
+The Python version used is 3.8.
+
+```shell
+$ mkvirtualenv -p /path/to/python3.8 rest-api
+```
+
+Install the required Python packages
+
+```shell
+pip install -r backend-api-server/requirements.txt
+```
+
+PostgreSQL 10 is used as database. Create a database and change the `backend-api-server/swagger_server/mals/db/handler/db_handler.py` accordingly.
+
+The database initializiation is done by calling init-mal2-db.sh
+```shell
+#!/bin/bash
+set -e
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
+ CREATE ROLE mal2user WITH LOGIN PASSWORD 'change_pass' SUPERUSER INHERIT CREATEDB CREATEROLE;
+ CREATE DATABASE mal2restdb OWNER mal2user;
+EOSQL
+```
+
+## Usage
+
+To run the server, please execute the following from the backend-api-server directory:
+
+```
+python3 -m swagger_server
+```
+
+and point your browser to:
+
+```
+http:https://localhost:8080/malzwei/ecommerce/1.1/ui/
+```
+
+Your Swagger definition lives here:
+
+```
+http:https://localhost:8080/malzwei/ecommerce/1.1/swagger.json
+```
+
+Attention, this is for development only. In a production environment, for example, [uwsgi](https://uwsgi-docs.readthedocs.io/en/latest/WSGIquickstart.html) can be used with [apache2](http:https://httpd.apache.org/).
+
+## Running with Docker
+
+For a full local deployment with an external db launch the docker build:
+```shell
+# starting the containers
+docker-compose -f docker-compose.local_dev.yml up
+```
+
+For a full server deployment launch the docker build:
+```shell
+# building the image
+docker-compose build --build-arg ENDPOINT_BASE=your.server.location
+
+# starting the containers
+docker-compose up
+```
+
+All details on the docker stand-alon or docker-compose build are provided in the files
+`docker-compose.yml`
+`docker-compose.local_dev.yml`
+and
+`backend-api-server/docker/Dockerfile`
+
+The code as is contains settings for local dev deployment. The Dockerfile uses 'sed' to provide the production configuration and docker-compose up to start the integrated system
+`RUN sed -i "s|127.0.0.1|$env_ENDPOINT_BASE|g" swagger_server/swagger/swagger.yaml`
+
+## REST-API
+
+The REST API documentation is available at http:https://localhost:8081/malzwei/ecommerce/1.1/ui/.
+
+Please note all port configurations of the server are defined within the docker-compose build stage. Therefore make sure when adjusting the ports within docker-compose.yml to have identical ports for container and host within the mal2-rest-api service as they are passed to connexion[swagger-ui] at build time.
+
+## About MAL2
+
+The MAL2 project applies Deep Neural Networks and Unsupervised Machine Learning to advance cybercrime prevention by a) automating the discovery of fraudulent eCommerce and b) detecting Potentially Harmful Apps (PHAs) in Android.
+The goal of the MAL2 project is to provide (i) an Open Source framework and expert tools with integrated functionality along the required pipeline – from malicious data archiving, feature selection and extraction, training of Machine Learning classification and detection models towards explainability in the analysis of results (ii) to execute its components at scale and (iii) to publish an annotated Ground-Truth dataset in both application domains. To raise awareness for cybercrime prevention in the general public, two demonstrators, a Fake-Shop Detection Browser Plugin as well as a Android Malware Detection Android app are released that allow live-inspection and AI based predictions on the trustworthiness of eCommerce sites and Android apps.
+
+The work is based on results carried out in the research project [MAL2 project](https://projekte.ffg.at/projekt/3044975), which was partially funded by the Austrian Federal Ministry for Climate Action, Environment, Energy, Mobility, Innovation and Technology (BMK) through the ICT of the future research program (6th call) managed by the Austrian federal funding agency (FFG).
+* Austrian Institute of Technology GmbH, Center for Digital Safety and Security [AIT](https://www.ait.ac.at/)
+* Austrian Institute for Applied Telecommunications [ÖIAT](https://www.oiat.at)
+* X-NET Services GmbH [XNET](https://x-net.at/de/)
+* Kuratorium sicheres Österreich [KSÖ](https://kuratorium-sicheres-oesterreich.at/)
+* IKARUS Security Software [IKARUS](https://www.ikarussecurity.com/)
+
+More information is available at [www.malzwei.at](http:https://www.malzwei.at)
+
+## Contact
+For details on behalf of the MAL2 consortium contact: 
+Andrew Lindley (project lead)
+Research Engineer, Data Science & Artificial Intelligence
+Center for Digital Safety and Security, AIT Austrian Institute of Technology GmbH
+Giefinggasse 4 | 1210 Vienna | Austria
+T +43 50550-4272 | M +43 664 8157848 | F +43 50550-4150
+[email protected] | www.ait.ac.at
+or
+Woflgang Eibner, X-NET Services GmbH, [email protected]
+
+## License
+The MAL2 Software stack is dual-licensed under commercial and open source licenses. 
+The Software in this repository is subject of the terms and conditions defined in file 'LICENSE.md'

backend-api-server/.dockerignore

@@ -0,0 +1,78 @@
+.travis.yaml
+.swagger-codegen-ignore
+README.md
+tox.ini
+git_push.sh
+test-requirements.txt
+setup.py
+
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+venv/
+.python-version
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+#Ipython Notebook
+.ipynb_checkpoints
+
+#mal2-generated stuff
+logs/
+.vscode
+.empty
+

backend-api-server/.gitignore

@@ -0,0 +1,64 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+# Usually these files are written by a python script from a template
+# before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+venv/
+.python-version
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+#Ipython Notebook
+.ipynb_checkpoints

backend-api-server/.swagger-codegen-ignore

@@ -0,0 +1,23 @@
+# Swagger Codegen Ignore
+# Generated by swagger-codegen https://github.com/swagger-api/swagger-codegen
+
+# Use this file to prevent files from being overwritten by the generator.
+# The patterns follow closely to .gitignore or .dockerignore.
+
+# As an example, the C# client generator defines ApiClient.cs.
+# You can make changes and tell Swagger Codgen to ignore just this file by uncommenting the following line:
+#ApiClient.cs
+
+# You can match any string of characters against a directory, file or extension with a single asterisk (*):
+#foo/*/qux
+# The above matches foo/bar/qux and foo/baz/qux, but not foo/bar/baz/qux
+
+# You can recursively match patterns against a directory, file or extension with a double asterisk (**):
+#foo/**/qux
+# This matches foo/bar/qux, foo/baz/qux, and foo/bar/baz/qux
+
+# You can also negate patterns with an exclamation (!).
+# For example, you can ignore all files in a docs folder with the file extension .md:
+#docs/*.md
+# Then explicitly reverse the ignore rule for a single file:
+#!docs/README.md

backend-api-server/.swagger-codegen/VERSION

@@ -0,0 +1 @@
+3.0.23

backend-api-server/.travis.yml

@@ -0,0 +1,13 @@
+# ref: https://docs.travis-ci.com/user/languages/python
+language: python
+python:
+ - "3.2"
+ - "3.3"
+ - "3.4"
+ - "3.5"
+ #- "3.5-dev" # 3.5 development branch
+ #- "nightly" # points to the latest development branch e.g. 3.6-dev
+# command to install dependencies
+install: "pip install -r requirements.txt"
+# command to run tests
+script: nosetests

backend-api-server/docker/Dockerfile

@@ -0,0 +1,54 @@
+FROM python:3.7.6-buster
+
+#Sets the APIs public endpoint default-location to localhost
+#override by calling docker-compose build --build-arg ENDPOINT_BASE=mal2.ait.ac.at
+ARG ENDPOINT_BASE=localhost
+ARG ENDPOINT_PORT=8080
+RUN echo "building MAL2-REST API public endpoint: '$ENDPOINT_BASE' port: '$ENDPOINT_PORT'"
+
+#our working dir in the container
+RUN mkdir -p /mal2/rest-api
+WORKDIR /mal2/rest-api
+
+# need to copy as volumens are only accessible once container created
+COPY ./backend-api-server/. /mal2/rest-api
+
+#install our mal2-model dependency
+RUN pip3 install docker/kosoh_logo_classifier-0.1-py3-none-any.whl
+RUN pip3 install docker/mal2_fakeshop_models-0.25-py3-none-any.whl
+
+#install swagger-ui which allows browsing open-api app routes e.g. http:https://localhost:8080/malzwei/ecommerce/1.1/ui/
+RUN pip3 install connexion[swagger-ui]
+
+#install all other dependencies for the mal2-rest app
+RUN pip3 install --no-cache-dir -r requirements.txt
+
+#replace the mal2 apps db configuration (port and host) using sed
+RUN sed -i 's/54320/5432/g' swagger_server/mal2/db/handler/db_handler.py
+RUN sed -i 's/127.0.0.1/mal2-fakeshop-plugin_rest-api_db/g' swagger_server/mal2/db/handler/db_handler.py
+
+#ENV vars are available to container (also beyond build)
+ENV env_ENDPOINT_BASE=$ENDPOINT_BASE
+ENV env_ENDPOINT_PORT=$ENDPOINT_PORT
+
+#replace API public endpoint location. [used by swagger-ui]
+RUN sed -i "s|127.0.0.1|$env_ENDPOINT_BASE|g" swagger_server/swagger/swagger.yaml
+RUN sed -i "s|8080|$env_ENDPOINT_PORT|g" swagger_server/swagger/swagger.yaml
+RUN sed -i "s|8080|$env_ENDPOINT_PORT|g" swagger_server/__main__.py
+
+#remove data import limits i.e. import all 
+RUN sed -i "s|limit_imported_items=10||g" swagger_server/mal2/db/handler/db_whitelist_handler.py
+RUN sed -i "s|limit_imported_items=10||g" swagger_server/mal2/db/handler/db_ignorelist_handler.py
+RUN sed -i "s|limit_imported_items=10||g" swagger_server/mal2/db/handler/db_blacklist_handler.py
+RUN sed -i "s|limit_imported_items=10||g" swagger_server/mal2/db/handler/db_greylist_handler.py
+RUN sed -i "s|limit_imported_items=10||g" swagger_server/mal2/db/handler/db_handler.py
+
+#for debugging container interactively
+#ENTRYPOINT ["sh", "-c", "sleep 2073600"]
+
+EXPOSE $ENDPOINT_PORT
+
+ENTRYPOINT ["python3"]
+
+CMD ["-m", "swagger_server"]
+

backend-api-server/init-mal2-db.sh

@@ -0,0 +1,8 @@
+#!/bin/bash
+set -e
+
+psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname "$POSTGRES_DB" <<-EOSQL
+ CREATE ROLE mal2user WITH LOGIN PASSWORD 'change_pass' SUPERUSER INHERIT CREATEDB CREATEROLE;
+ CREATE DATABASE mal2restdb OWNER mal2user;
+EOSQL
+