add machine-id to all "local" indexer-requests & refactor indexer-request headers #17
Conversation
.env.dist
Outdated
| HUBGREP_CRAWLERS_USER_AGENT_SUFFIX="@ https://hubgrep.io/about" | ||
| HUBGREP_CRAWLERS_MACHINE_ID=no-id |
There was a problem hiding this comment.
tried to do some fancy uuid auto-generating in docker-compose.yml instead of manually setting it - but I couldnt get it to run =(
maybe you know how to do it?
this should work on linux/osx as a bash one-liner:
od -x /dev/urandom | head -1 | awk '{OFS="-"; print $2$3,$4,$5,$6,$7$8$9}'
otherwise this requires a dependancy and exported into the envvar:
uuid=$(uuidgen)
both found here:
https://serverfault.com/questions/103359/how-to-create-a-uuid-in-bash
There was a problem hiding this comment.
but its probably better not to anyway, for whenever we restart stuff etc.
| "X-Request-ID": crawler_uuid | ||
| "User-Agent": current_app.config["USER_AGENT"], | ||
| "X-Correlation-ID": crawler_uuid, # specific crawler | ||
| "Hubgrep-Crawler-Machine-ID": current_app.config["MACHINE_ID"] # shared by "local" crawlers (all hosters) |
There was a problem hiding this comment.
I misunderstood the "X-Request-ID" previously, and it only there to track the specific unique request (ofc.).
"X-Correlation-ID" is instead a header that is intended to be used across many requests to "follow the red thread" - this is our specific crawler instance, which we should use in debugging logs. Crawler uuid is still a bit broader than what the header intends as a "cross services transaction id" kind of thing, but I think it fits for us. Even if the same id comes up again and again.
As for a higher global id, I cant find any existing headers so I made our own "Hubgrep-Crawler-Machine-ID" which we will use to map api_keys from.
| @@ -19,6 +20,7 @@ | |||
|
|
|||
| def _hoster_session_request(method, session, url, error_count=0, *args, **kwargs): | |||
| try: | |||
| session.headers.update({"X-Request-ID": uuid.uuid4().hex}) | |||
There was a problem hiding this comment.
update this header on each request to the indexer
| platform_type = platform_data["type"] | ||
| api_url = platform_data["api_url"] | ||
| api_key = platform_data["api_key"] | ||
| api_key = platform_data.get("api_key", None) |
There was a problem hiding this comment.
In general, giving out an api_key is optional across all crawlers (but specific crawlers should throw if they require it)
This has to do with they way we assign api_keys, and that if you keep adding machines, they shouldnt share api_keys and thus once we run out of api_keys "None" will be used.
No description provided.