Skip to content

Commit

Permalink
Merge branch 'master' into feature/6_add_Docker_support
Browse files Browse the repository at this point in the history
  • Loading branch information
liadmagen committed Jul 23, 2018
2 parents 07296a9 + 4ef2109 commit 1ccf128
Show file tree
Hide file tree
Showing 6 changed files with 36 additions and 9 deletions.
2 changes: 1 addition & 1 deletion .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
name: run tests
command: |
. venv/bin/activate
pytest
pytest --cov=./
- store_artifacts:
path: test-reports
Expand Down
4 changes: 3 additions & 1 deletion .travis.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,6 @@ install:
script:
- py.test
notifications:
slack: vdsg:YeRk6YBgJuTEtaRT1pPp76pb
slack: vdsg:YeRk6YBgJuTEtaRT1pPp76pb
after_success:
- codecov --token 80bcc8a7-2f30-4656-8565-709d5af5b8d0
12 changes: 5 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,12 @@
# Keep-Current - The Web Miner

<!-- Badges section here. -->
[![Codacy Badge](https://api.codacy.com/project/badge/Grade/883c8e93b4934566b9dfdc6b91fa85e7)](https://app.codacy.com/app/Keep-Current/web-miner?utm_source=github.com&utm_medium=referral&utm_content=Keep-Current/web-miner&utm_campaign=badger)
[![Build Status](https://travis-ci.org/Keep-Current/web-miner.svg?branch=master)](https://travis-ci.org/Keep-Current/web-miner)
[![CircleCI](https://circleci.com/gh/Keep-Current/web-miner.svg?style=svg)](https://circleci.com/gh/Keep-Current/web-miner)
[![BCH compliance](https://bettercodehub.com/edge/badge/Keep-Current/web-miner?branch=master)](https://bettercodehub.com/)
[![codecov](https://codecov.io/gh/Keep-Current/web-miner/branch/master/graph/badge.svg)](https://codecov.io/gh/Keep-Current/web-miner)
[![codebeat badge](https://codebeat.co/badges/03da69a3-74cf-468d-80f9-bc62651323f7)](https://codebeat.co/projects/github-com-keep-current-web-miner-master)

## Web Miner

Expand All @@ -15,11 +18,6 @@ For example, for [arxiv.org](https://arxiv.org/help/api/index).

### Potential tools to implement

## Web Miner

This repository deploys a web spider and documenr miner, that given a specific set of sources (URLs), should locate new documents (web-pages) and save them in the DB for future processing.
When possible, in websites that allow, an API can be used. For example, for [arxiv.org](https://arxiv.org/help/api/index).

We lean heavily on existing tools as well as developing our own new methods.

- [scrapy](https://scrapy.org/) which later we hope to host on [scrapy-cloud](https://scrapinghub.com/scrapy-cloud)
Expand Down Expand Up @@ -80,9 +78,9 @@ flask run

We follow the [clean architecture style](https://blog.thedigitalcatonline.com/blog/2016/11/14/clean-architectures-in-python-a-step-by-step-example/) and structure the codebase accordingly.

![ceanArchitecture image](https://cdn-images-1.medium.com/max/1600/1*B7LkQDyDqLN3rRSrNYkETA.jpeg)
![cleanArchitecture image](https://cdn-images-1.medium.com/max/1600/1*B7LkQDyDqLN3rRSrNYkETA.jpeg)

*Image creadit to [Thang Chung under MIT terms](https://github.com/thangchung/blog-core)*
*Image credit to [Thang Chung under MIT terms](https://github.com/thangchung/blog-core)*

## Who are we?

Expand Down
26 changes: 26 additions & 0 deletions codecov.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
codecov:
notify:
require_ci_to_pass: yes

coverage:
precision: 2
round: down
range: "70...100"

status:
project: yes
patch: yes
changes: no

parsers:
gcov:
branch_detection:
conditional: yes
loop: yes
method: no
macro: no

comment:
layout: "header, diff"
behavior: default
require_changes: no
1 change: 1 addition & 0 deletions requirements/test.txt
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ tox
coverage
pytest-cov
pytest-flask
codecov
gunicorn
flake8
Sphinx
Empty file added tests/tox.ini
Empty file.

0 comments on commit 1ccf128

Please sign in to comment.