Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Removed kafka integration, replaced by webhooks #27

Merged
merged 21 commits into from
Nov 21, 2022
Merged

Removed kafka integration, replaced by webhooks #27

merged 21 commits into from
Nov 21, 2022

Conversation

quaxsze
Copy link
Contributor

@quaxsze quaxsze commented Oct 25, 2022 •
edited by abulte

@quaxsze quaxsze changed the title Removed kafka integration, replaced by webhooks WIP: Removed kafka integration, replaced by webhooks Oct 25, 2022
@quaxsze quaxsze requested a review from maudetes October 25, 2022 15:41
@quaxsze
Copy link
Contributor Author

quaxsze commented Oct 25, 2022

Still work in progress but I'd like a first review @maudetes

maudetes
maudetes reviewed Nov 15, 2022
View reviewed changes
Copy link
Contributor

@maudetes maudetes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A first batch of review.

But I think I had the same logic in mind :)

udata_hydra/utils/http.py Outdated Show resolved Hide resolved
udata_hydra/crawl.py Outdated Show resolved Hide resolved
abulte
abulte reviewed Nov 16, 2022
View reviewed changes
Copy link
Contributor

@abulte abulte left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if we need a Celery or RQ to handle http calls to udata. Using aiohttp instead of requests will make it "less blocking" but still.

We'll need one scheduler anyway when adding analysis. But maybe we can wait till then.

udata_hydra/utils/http.py Outdated Show resolved Hide resolved
@quaxsze quaxsze changed the title WIP: Removed kafka integration, replaced by webhooks Removed kafka integration, replaced by webhooks Nov 16, 2022
maudetes
maudetes reviewed Nov 17, 2022
View reviewed changes
Copy link
Contributor

@maudetes maudetes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some comments, notably on the relation with udata

udata_hydra/utils/http.py Outdated Show resolved Hide resolved
udata_hydra/utils/http.py Outdated Show resolved Hide resolved
udata_hydra/utils/http.py Show resolved Hide resolved
udata_hydra/datalake_service.py Outdated Show resolved Hide resolved
udata_hydra/crawl.py Outdated Show resolved Hide resolved
udata_hydra/datalake_service.py Outdated Show resolved Hide resolved
udata_hydra/datalake_service.py Outdated Show resolved Hide resolved
udata_hydra/utils/http.py Outdated Show resolved Hide resolved
udata_hydra/utils/http.py Outdated Show resolved Hide resolved
quaxsze and others added 15 commits November 17, 2022 14:19
@quaxsze @maudetes
Update udata_hydra/utils/http.py
7bf5d44 
Co-authored-by: maudetes <[email protected]>
@quaxsze @maudetes
Update udata_hydra/datalake_service.py
cfa23b6 
Co-authored-by: maudetes <[email protected]>
@quaxsze @maudetes
Update udata_hydra/crawl.py
1394806 
Co-authored-by: maudetes <[email protected]>
@quaxsze @maudetes
Update udata_hydra/datalake_service.py
9834baf 
Co-authored-by: maudetes <[email protected]>
@quaxsze @maudetes
Update udata_hydra/datalake_service.py
733f43a 
Co-authored-by: maudetes <[email protected]>
@quaxsze @maudetes
Update udata_hydra/utils/http.py
99075c7 
Co-authored-by: maudetes <[email protected]>
@quaxsze @maudetes
Update udata_hydra/utils/http.py
d024d80 
Co-authored-by: maudetes <[email protected]>
@maudetes
Fix typo
f1f8181
@quaxsze
add requests validation and tests
c17c203
@quaxsze
Merge branch 'Kafka-eventectomie' of github.com:etalab/udata-hydra in…
191a71a 
…to Kafka-eventectomie
@quaxsze
tests
1281041
@maudetes @quaxsze
Add minimal config to save to minio or not
9ad3980
@quaxsze
tests
95315af
@maudetes
Allow none in resource document validation and remove None extras
5ecaba7
@maudetes
Fix lint
b364198
maudetes
maudetes approved these changes Nov 21, 2022
View reviewed changes
Copy link
Contributor

@maudetes maudetes left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh yeah! let's merge! 🚀

udata_hydra/utils/http.py Show resolved Hide resolved
@quaxsze quaxsze merged commit fed48ab into main Nov 21, 2022
@quaxsze quaxsze deleted the Kafka-eventectomie branch November 21, 2022 17:36
@maudetes maudetes mentioned this pull request Jul 22, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@abulte abulte abulte left review comments

@maudetes maudetes maudetes approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
3 participants
@quaxsze @abulte @maudetes