Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug] [API] make pagination optional for GET /api/user/repos #11800

Closed
6543 opened this issue Jun 8, 2020 · 18 comments · Fixed by #11827
Closed

[Bug] [API] make pagination optional for GET /api/user/repos #11800

6543 opened this issue Jun 8, 2020 · 18 comments · Fixed by #11827
Labels
issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP
Milestone

Comments

@6543
Copy link
Member

6543 commented Jun 8, 2020

come up with #9452 in v1.12
ref #9452 (comment)

@lafriks lafriks added this to the 1.12.0 milestone Jun 8, 2020
@lafriks lafriks added the issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP label Jun 8, 2020
@CirnoT
Copy link
Contributor

CirnoT commented Jun 8, 2020

Disagree, the mentioned comment clearly points at issue in Drone implementation which is missing pagination enumeration. We don't explicitly require pagination options to be set and with or without we need some sane limit on what we return. The instance might have 10 repos or it might have 2000 repos.

@6543
Copy link
Member Author

6543 commented Jun 8, 2020

I'd like to have both

@6543
Copy link
Member Author

6543 commented Jun 8, 2020

-> I'll take the gitea part

@CirnoT
Copy link
Contributor

CirnoT commented Jun 8, 2020

We definitely should NOT return all repos on API, instances can be small or can be huge.

This is not a bug but breaking change on our part since we added pagination; tools should now be updated to use it properly.

@lafriks
Copy link
Member

lafriks commented Jun 8, 2020

@CirnoT yeah but we can not just break drone integration. We can add deprecation notice and ask for drone developers to use paging and later in next version drop that. I think drone integration is used quite extensively to just break it

@techknowlogick
Copy link
Member

@lafriks the recent PR by @CirnoT provides a good way forward as it relies on app.ini setting for default response size, as if we don't have pagination then users with many repos could DOS themselves and run out of memory.

@6543
Copy link
Member Author

6543 commented Jun 8, 2020

-> #11808

and for the DOS at witch point it get critical, we could add a hardcoded lim at 50/100/200 ...

@6543
Copy link
Member Author

6543 commented Jun 8, 2020

sadly I dont know how to "depricate" a optional value to be required ...

think the best way is just inform huge project that they have to implement paggination for v1.13 now and switch in v1.13 ?

@6543
Copy link
Member Author

6543 commented Jun 8, 2020

#11805 ...

@lafriks
Copy link
Member

lafriks commented Jun 8, 2020

@techknowlogick Yes but currently we do return all repos in versions <=v1.11

@CirnoT
Copy link
Contributor

CirnoT commented Jun 8, 2020

I think 30 is a sane default till a fix is pushed to go-scm for pagination support. People that can not update Drone for some reason will be able to adjust the default and max limits to their needs.

@lafriks
Copy link
Member

lafriks commented Jun 8, 2020

At my company git I have about 100 repos so that will break drone integration for me ;)

@lafriks
Copy link
Member

lafriks commented Jun 8, 2020

And as not everyone read changelog it will make a lot users not knowing why repos is not syncing anymore in drone and most probably they will complain to drone devs

@lafriks
Copy link
Member

lafriks commented Jun 9, 2020

To give a point it is already creating confusion, see article comment: https://dev.to/ruanbekker/self-hosted-cicd-with-gitea-and-drone-ci-200l

@techknowlogick
Copy link
Member

techknowlogick commented Jun 9, 2020

@lafriks with @CirnoT's PR you can change settings to increase value to 100 so that all your repos show up. All we can do is to put out the correct information, and send corrections to articles we see wrong. There are so many Gitea articles that say to wget version 1.2 of Gitea 😬

@lafriks
Copy link
Member

lafriks commented Jun 9, 2020

I know that and I will do that but for many this will cause confusion, someone should really send PR to fix drone/gitea integration before we release 1.12

@ptman
Copy link
Contributor

ptman commented Jun 10, 2020

Are the drone developers even aware of this ongoing discussion? I too feel like deprecation first and removal later is the right approach. Maybe allow turning off compat for deprecated features in config?

@lafriks
Copy link
Member

lafriks commented Jun 11, 2020

@ptman there is already PR that will fix drone support so that no changes in drone are needed

zeripath pushed a commit that referenced this issue Jun 13, 2020
* Add count to `GetUserRepositories` so that pagination can be supported for `/user/{username}/repos`
* Rework ListMyRepos to use models.SearchRepository

ListMyRepos was an odd one. It first fetched all user repositories and then tried to supplement them with accessible map. The end result was that:

* Limit for pagination did not work because accessible repos would always be appended
* The amount of pages was incorrect if one were to calculate it
* When paginating, all accessible repos would be shown on every page

Hopefully it should now work properly. Fixes #11800 and does not require any change on Drone-side as it can properly interpret and act on Link header which we now set.

Co-authored-by: Lauris BH <[email protected]>
CirnoT added a commit to CirnoT/gitea that referenced this issue Jun 13, 2020
* Add count to `GetUserRepositories` so that pagination can be supported for `/user/{username}/repos`
* Rework ListMyRepos to use models.SearchRepository

ListMyRepos was an odd one. It first fetched all user repositories and then tried to supplement them with accessible map. The end result was that:

* Limit for pagination did not work because accessible repos would always be appended
* The amount of pages was incorrect if one were to calculate it
* When paginating, all accessible repos would be shown on every page

Hopefully it should now work properly. Fixes go-gitea#11800 and does not require any change on Drone-side as it can properly interpret and act on Link header which we now set.

Co-authored-by: Lauris BH <[email protected]>
(cherry picked from commit 0159851)
zeripath pushed a commit that referenced this issue Jun 13, 2020
* Add count to `GetUserRepositories` so that pagination can be supported for `/user/{username}/repos`
* Rework ListMyRepos to use models.SearchRepository

ListMyRepos was an odd one. It first fetched all user repositories and then tried to supplement them with accessible map. The end result was that:

* Limit for pagination did not work because accessible repos would always be appended
* The amount of pages was incorrect if one were to calculate it
* When paginating, all accessible repos would be shown on every page

Hopefully it should now work properly. Fixes #11800 and does not require any change on Drone-side as it can properly interpret and act on Link header which we now set.

Co-authored-by: Lauris BH <[email protected]>
(cherry picked from commit 0159851)
ydelafollye pushed a commit to ydelafollye/gitea that referenced this issue Jul 31, 2020
* Add count to `GetUserRepositories` so that pagination can be supported for `/user/{username}/repos`
* Rework ListMyRepos to use models.SearchRepository

ListMyRepos was an odd one. It first fetched all user repositories and then tried to supplement them with accessible map. The end result was that:

* Limit for pagination did not work because accessible repos would always be appended
* The amount of pages was incorrect if one were to calculate it
* When paginating, all accessible repos would be shown on every page

Hopefully it should now work properly. Fixes go-gitea#11800 and does not require any change on Drone-side as it can properly interpret and act on Link header which we now set.

Co-authored-by: Lauris BH <[email protected]>
@go-gitea go-gitea locked and limited conversation to collaborators Nov 24, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
issue/critical This issue should be fixed ASAP. If it is a PR, the PR should be merged ASAP
Projects
None yet
5 participants