Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revisit Repository Syncing #2036

Closed
bradrydzewski opened this issue May 11, 2017 · 0 comments
Closed

Revisit Repository Syncing #2036

bradrydzewski opened this issue May 11, 2017 · 0 comments
Projects
Milestone

Comments

@bradrydzewski
Copy link

Early versions of Drone would sync the user repositories and permissions with the database. This was slow and was problematic for certain providers (Bitbucket) which had certain API restrictions. I therefore decided in 0.5 to remove the sync and fetch the repository list and permissions and cache in ram.

Most Drone users probably think this works quite well, however, there have been some unforseen issues with individuals that have access to thousands of repositories:

  1. Drone is required to make many (20+) api calls for 1000+ repositories, which can be slow
  2. In addition to being slow it can cause users to max out their API limits
  3. The implementation requires the IN clause which is limited to the number of input parameters

There have also been some technology improvements that can simplify syncing. Namely the fact that Postgres (as well as other vendors) now all support the INSERT IGNORE syntax.

Bitbucket and Bitbucket server are still a problem. I believe we can solve this by storing a TTL for the individual repository permissions. When the TTL expires we make an API call. This prevents us from having to update thousands of rows of permissions on sync.

This should also address some UX issues like https://github.com/drone/drone-ui/issues/79

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
No open projects
Development

No branches or pull requests

1 participant