Unpin fsspec

[merelcht]

Description

Fixes #3102

Development notes

In the newer fsspec versions permissions are checked at a different stage so the following line already threw an error:
The AWS Access Key Id you provided does not exist in our records.

https://github.com/kedro-org/kedro/blob/main/kedro/io/core.py#L539

version_paths = sorted(self._glob_function(pattern), reverse=True)

I added a try except to handle this.

Developer Certificate of Origin

We need all contributions to comply with the Developer Certificate of Origin (DCO). All commits must be signed off by including a Signed-off-by line in the commit message. See our wiki for guidance.

If your PR is blocked due to unsigned commits, then you must follow the instructions under "Rebase the branch" on the GitHub Checks page for your PR. This will retroactively add the sign-off to all unsigned commits and allow the DCO check to pass.

Checklist

• Read the contributing guidelines
• Signed off each commit with a Developer Certificate of Origin (DCO)
• Opened this PR as a 'Draft Pull Request' if it is work-in-progress
• Updated the documentation to reflect the code changes
• Added a description of this change in the RELEASE.md file
• Added tests to cover my changes
• Checked if this change will affect Kedro-Viz, and if so, communicated that with the Viz team

[noklam]

Why do we want to catch the error if it explicitly saying it's a permission error?

[merelcht]

Why do we want to catch the error if it explicitly saying it's a permission error?

My thinking was that fsspec is used for all filesystem types so it could be a different error is thrown if it's not a cloud system.

[AhdraMeraliQB]

LGTM! Just a QQ:

[AhdraMeraliQB]

Is the new error handling compatible with 2021.4<=version<2023.9?

[merelcht]

In the older versions the error will be caught later on in this check https://github.com/kedro-org/kedro/blob/main/kedro/io/core.py#L544

It's not possible to keep it fully backwards compatible, because the test-coverage would then fail and in different ways depending on which version of fsspec is installed. It's just a test-requirement though, so backward compatibility isn't as important as for the regular dependencies.

[AhdraMeraliQB]

Could we then pin the test requirement at >2023.9?

[merelcht]

I ended up just removing fsspec from the test requirements, because it wasn't included before the pin was needed.

[SajidAlamQB]

LGTM 👍