-
Notifications
You must be signed in to change notification settings - Fork 187
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Py deps upgrade, TF upgrade, TF fixes #174
Open
phretor
wants to merge
12
commits into
airbnb:master
Choose a base branch
from
phretor:master
base: master
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
12 commits
Select commit
Hold shift + click to select a range
f7bde9d
Merging some pending PRs
phretor 2ad0419
python3 -> python
phretor 3545bb4
Added setters/getters for aws_account_name
phretor fa55d56
Added ${var.aws_account_name} to kms.tf
phretor adb84dd
Added `aws_account_name` to variables.tf
phretor ef1db81
Upgraded python deps
phretor 84aabc9
TF 0.12.9 -> 0.14.5
phretor 3dc3fee
TF 0.13 -> 0.14 constrain
phretor 295857d
Lambda runtime from py36 to py37
phretor 7472b0c
Automated Lambda deps build
phretor e2c7149
Updated Lambda README.rst
phretor b2453e6
New Lambda deps ZIP file
phretor File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,117 @@ | ||
FROM amazonlinux:latest | ||
|
||
ENV APP_DIR /var/task | ||
ENV LAMBDA_DIR ${APP_DIR}/lambda | ||
ENV PIP_DIR $APP_DIR/pip | ||
|
||
ENV CRYPTO_VER 3.3.1 | ||
ENV ASN1CRYPTO_VER 1.4.0 | ||
ENV YARA_VER 4.0.0 | ||
ENV UPX_VER 3.94 | ||
ENV YEXTEND_VER 1.6 | ||
ENV ARCH amd64 | ||
|
||
VOLUME /var/out | ||
WORKDIR $APP_DIR | ||
|
||
RUN yum -y update \ | ||
&& yum -y install \ | ||
git \ | ||
zip \ | ||
xz \ | ||
wget \ | ||
tar \ | ||
install \ | ||
autoconf \ | ||
automake \ | ||
pkgconfig \ | ||
bzip2-devel \ | ||
gcc \ | ||
gcc-c++ \ | ||
libarchive-devel \ | ||
libffi-devel \ | ||
libtool \ | ||
libuuid-devel \ | ||
make \ | ||
openssl-devel \ | ||
pcre-devel \ | ||
poppler-utils \ | ||
poppler-cpp-devel \ | ||
zlib-devel \ | ||
python3 \ | ||
python3-pip \ | ||
python3-devel \ | ||
&& yum clean all \ | ||
&& pip3 install nose | ||
|
||
# install Yara | ||
RUN wget -O yara.tar.gz https://github.com/VirusTotal/yara/archive/v${YARA_VER}.tar.gz \ | ||
&& tar -xzf yara.tar.gz \ | ||
&& cd yara-${YARA_VER} \ | ||
&& ./bootstrap.sh \ | ||
&& ./configure \ | ||
&& make \ | ||
&& make check \ | ||
&& make install \ | ||
&& echo "/usr/local/lib" > /etc/ld.so.conf.d/yara.conf \ | ||
&& ldconfig \ | ||
&& yara -v | ||
|
||
# Install Python pacakges | ||
RUN pip3 install \ | ||
"cryptography==${CRYPTO_VER}" \ | ||
"yara-python==${YARA_VER}" \ | ||
"asn1crypto==${ASN1CRYPTO_VER}" \ | ||
-t $PIP_DIR | ||
|
||
# Copy crypto libs | ||
RUN cd $PIP_DIR \ | ||
&& rm -r *.dist-info *.egg-info \ | ||
&& find . -name __pycache__ | xargs rm -rf \ | ||
&& mv _cffi_backend.cpython-*m-x86_64-linux-gnu.so _cffi_backend.so \ | ||
&& cd cryptography/hazmat/bindings \ | ||
&& mv _openssl.abi3.so _openssl.so \ | ||
&& mv _padding.abi3.so _padding.so | ||
|
||
# Gather pip files | ||
RUN mkdir ${LAMBDA_DIR} \ | ||
&& cp -r ${PIP_DIR}/* ${LAMBDA_DIR} | ||
|
||
# Compile yextend | ||
RUN mkdir ${LAMBDA_DIR}/libs \ | ||
&& git clone https://github.com/BayshoreNetworks/yextend.git \ | ||
&& cd yextend \ | ||
&& git checkout yara-${YARA_VER} \ | ||
&& mv configure.ac configure.ac.orig \ | ||
&& sed 's/python \-c/python3 -c/g' configure.ac.orig > configure.ac \ | ||
&& ./build.sh \ | ||
&& make unittests \ | ||
&& cp yextend ${LAMBDA_DIR} \ | ||
&& cp libs/*.o ${LAMBDA_DIR}/libs \ | ||
&& cp libs/*.yara ${LAMBDA_DIR}/libs | ||
|
||
# UPX | ||
RUN wget https://github.com/upx/upx/releases/download/v${UPX_VER}/upx-${UPX_VER}-${ARCH}_linux.tar.xz \ | ||
&& tar -xf upx-${UPX_VER}-${ARCH}_linux.tar.xz \ | ||
&& cp upx-${UPX_VER}-${ARCH}_linux/upx ${LAMBDA_DIR} | ||
|
||
## Gather compiled libraries | ||
RUN cp /usr/bin/pdftotext ${LAMBDA_DIR} \ | ||
&& mkdir tmplib \ | ||
&& find \ | ||
/usr/lib64/lib* \ | ||
-depth \ | ||
-type f \ | ||
-o \ | ||
-type d \ | ||
-name 'lib*.so.*' | \ | ||
grep -E 'lib(archive|fontconfig|jbig|jpeg|lcms|lzma|lzo2|openjpeg|pcrecpp|png|poppler|stdc|tiff|xml)' | \ | ||
cpio -pamVd tmplib \ | ||
&& mv tmplib/usr/lib64/* ${LAMBDA_DIR} \ | ||
&& cp /usr/local/lib/libyara.so.* ${LAMBDA_DIR} | ||
|
||
RUN cd ${LAMBDA_DIR} \ | ||
&& zip -r dependencies.zip * \ | ||
&& mv ${LAMBDA_DIR}/dependencies.zip ${APP_DIR}/dependencies.zip | ||
|
||
CMD cp ${APP_DIR}/dependencies.zip /var/out/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
all: | ||
docker build -t binaryalert-lambda . | ||
docker run --rm -it -v ${PWD}:/var/out binaryalert-lambda |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,111 +1,20 @@ | ||
YARA Analyzer | ||
============= | ||
This Lambda function is the core of BinaryAlert. Each invocation downloads one or more binaries from | ||
S3, scans them against all available YARA rules, and forwards any matches to Dynamo and SNS. | ||
This Lambda function is the core of BinaryAlert. Each invocation downloads one | ||
or more binaries from S3, scans them against all available YARA rules, and | ||
forwards any matches to Dynamo and SNS. | ||
|
||
|
||
Updating YARA Binaries | ||
---------------------- | ||
Many libraries used by BinaryAlert are natively compiled, and must therefore be pre-built on an | ||
Amazon Linux AMI in order to run in Lambda. This has already been done for you: | ||
``dependencies.zip`` contains the following pre-built libraries: | ||
Many libraries used by BinaryAlert are natively compiled, and must therefore be | ||
pre-built on an Amazon Linux AMI in order to run in Lambda. This has already | ||
been done for you in the `dependencies.zip` file that ships with the repo, but you | ||
can rebuild it yourself via Docker. | ||
|
||
- `cryptography <https://cryptography.io>`_ (v2.3) | ||
- `UPX <https://github.com/upx/upx>`_ (v3.94) | ||
- `yara-python <https://github.com/VirusTotal/yara-python>`_ (v3.8.0) | ||
- `yara <https://github.com/VirusTotal/yara>`_ (v3.8.0) | ||
- `yextend <https://github.com/BayshoreNetworks/yextend>`_ (v1.6) | ||
- `pdftotext <https://poppler.freedesktop.org/>`_ (v0.26.5) | ||
|
||
If, however, you need to update or re-create the zipfile, SSH to an EC2 instance running the | ||
`AWS Lambda AMI <https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html>`_ | ||
and install the dependencies as follows: | ||
If you need to update or re-create the ZIP file, do it before deployment. | ||
|
||
.. code-block:: bash | ||
$ make | ||
|
||
# Install requirements | ||
sudo yum update | ||
sudo yum install autoconf automake bzip2-devel gcc64 gcc64-c++ libarchive-devel libffi-devel \ | ||
libtool libuuid-devel openssl-devel pcre-devel poppler-utils python36 python36-devel zlib-devel | ||
sudo pip install nose | ||
|
||
# Compile YARA | ||
wget https://github.com/VirusTotal/yara/archive/v3.8.0.tar.gz | ||
tar -xzf v3.8.0.tar.gz | ||
cd yara-3.8.0 | ||
./bootstrap.sh | ||
./configure | ||
make | ||
make check # Run unit tests | ||
sudo make install | ||
|
||
# Install cryptography and yara-python | ||
cd ~ | ||
mkdir pip | ||
pip-3.6 install cryptography yara-python -t pip | ||
|
||
# Compile yextend | ||
wget https://github.com/BayshoreNetworks/yextend/archive/1.6.tar.gz | ||
tar -xvzf 1.6.tar.gz | ||
cd yextend-1.6 | ||
# Manually: modify main.cpp, line 473 to hardcode the yara version to 3.8 | ||
./build.sh | ||
make unittests # Run unit tests | ||
|
||
# Clean cryptography files | ||
cd ~/pip | ||
rm -r *.dist-info *.egg-info | ||
find . -name __pycache__ | xargs rm -r | ||
mv _cffi_backend.cpython-36m-x86_64-linux-gnu.so _cffi_backend.so | ||
cd cryptography/hazmat/bindings | ||
mv _constant_time.abi3.so _constant_time.so | ||
mv _openssl.abi3.so _openssl.so | ||
mv _padding.abi3.so _padding.so | ||
|
||
# Gather pip files | ||
cd ~ | ||
mkdir lambda | ||
cp pip/.libs_cffi_backend/* lambda | ||
cp -r pip/* lambda | ||
mv lambda/yara.cpython-36m-x86_64-linux-gnu.so lambda/yara.so | ||
wget https://raw.githubusercontent.com/VirusTotal/yara/master/COPYING -O lambda/YARA_LICENSE | ||
wget https://raw.githubusercontent.com/VirusTotal/yara-python/master/LICENSE -O lambda/YARA_PYTHON_LICENSE | ||
|
||
# Gather Yextend files | ||
cp yextend-1.6/yextend lambda | ||
cp yextend-1.6/LICENSE lambda/YEXTEND_LICENSE | ||
mkdir lambda/libs | ||
cp yextend-1.6/libs/*.o lambda/libs | ||
cp yextend-1.6/libs/*.yara lambda/libs | ||
|
||
# Download UPX | ||
wget https://github.com/upx/upx/releases/download/v3.94/upx-3.94-amd64_linux.tar.xz | ||
tar -xf upx-3.94-amd64_linux.tar.xz | ||
cp upx-3.94-amd64_linux/upx lambda | ||
cp upx-3.94-amd64_linux/COPYING lambda/UPX_LICENSE | ||
|
||
# Gather compiled libraries | ||
cp /usr/bin/pdftotext lambda | ||
cp /usr/lib64/libarchive.so.13 lambda | ||
cp /usr/lib64/libfontconfig.so.1 lambda | ||
cp /usr/lib64/libfreetype.so.6 lambda | ||
cp /usr/lib64/libjbig.so.2.0 lambda | ||
cp /usr/lib64/libjpeg.so.62 lambda | ||
cp /usr/lib64/liblcms2.so.2 lambda | ||
cp /usr/lib64/liblzma.so.5 lambda | ||
cp /usr/lib64/liblzo2.so.2 lambda | ||
cp /usr/lib64/libopenjpeg.so.2 lambda | ||
cp /usr/lib64/libpcrecpp.so.0 lambda | ||
cp /usr/lib64/libpng12.so.0 lambda | ||
cp /usr/lib64/libpoppler.so.46 lambda | ||
cp /usr/lib64/libstdc++.so.6 lambda | ||
cp /usr/lib64/libtiff.so.5 lambda | ||
cp /usr/lib64/libxml2.so.2 lambda | ||
cp /usr/local/lib/libyara.so.3 lambda | ||
|
||
# Build Zipfile | ||
cd lambda | ||
zip -r dependencies.zip * | ||
|
||
|
||
Then ``scp`` the ``dependencies.zip`` package to replace the one in the repo. | ||
And you'll find the new `dependencies.zip` file in this folder. |
Binary file not shown.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I remember right, this is an auto-generated file; did it change with the latest version of
sphinx
?