Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bugfix: user-agent python-requests #8394

Merged
merged 8 commits into from
Sep 11, 2023
Merged

Conversation

mtricolici98
Copy link
Contributor

@mtricolici98 mtricolici98 commented Sep 1, 2023

Replaced user agent to avoid bot protection on private repositories.

For lock and install commands, the requests library was used, resulting in HTTP requests with 'python-requests' User-agent being sent.

It is possible that HTTP Servers (such as OPNSens in my case) have 'bot protection' to automatically block "bot-like" user agents.

This quick fix comes as a counter measure for the following problem.

Pull Request Check List

Resolves: No tracked issue

  • No code tests updated
  • No documentation updates necessary

Copy link
Contributor

@dimbleby dimbleby left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this all seems a bit silly to me: a repository that doesn't allow downloads from the requests user agent is broken and should be fixed (or reconfigured)

however it also seems harmless enough I suppose

@@ -193,7 +195,8 @@ def authenticated_url(self, url: str) -> str:
def request(
self, method: str, url: str, raise_for_status: bool = True, **kwargs: Any
) -> requests.Response:
headers = kwargs.get("headers")
headers = kwargs.get("headers", {}) or {}
Copy link
Contributor

@dimbleby dimbleby Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

or {} isn't doing anything useful

cf

session.headers["User-Agent"] = self.user_agent
, which does this on the session - it would be consistent here to do the work in create_session().

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think or {} covers the potential of there being a headers key in the dict with a None value.

I will apply the UA to the session itself. Thank you for the suggestion.

@mtricolici98
Copy link
Contributor Author

this all seems a bit silly to me: a repository that doesn't allow downloads from the requests user agent is broken and should be fixed (or reconfigured)

however it also seems harmless enough I suppose

I do agree, it took me a while to figure out that the 403 I was receiving wasn't from bad credentials or authorization but from the user agent. It seems like our repository is proxied through a nginx configuration that blocks some "bot like" user agents, as to "protect" from vulnerability scans or other bot activity. A quick curl with 2 different user agent headers cleared that one for me. However the dev-ops team on my project is reluctant to disable the bot protection.
Thus I created a quite silly plugin to inject the ua headers, and thought it might be fine to also solve this at the source, especially since poetry provides his own user agent during the publishing step, it makes sense that for consistency it will also use the poetry UA.

@radoering radoering enabled auto-merge (squash) September 11, 2023 15:36
@radoering radoering merged commit a8987e3 into python-poetry:master Sep 11, 2023
17 checks passed
@dimbleby dimbleby mentioned this pull request Sep 29, 2023
2 tasks
Copy link

github-actions bot commented Mar 3, 2024

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Mar 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants