Rewrite git history

[andiradulescu]

This tackles #28840 and is a continuation of #30824.

This PR addresses master:

• prevent new large files stricter file size limits #30839
• move old objects, binary objects and big text files to LFS https://github.com/commaai/openpilot/pull/31562/files#diff-35f170270f2172a621c54f4e4ceb7f9675b74a90eb359b376202f6a80c819aa2R231
• prepend the devel release history - from v0.7.1
• tags - all tagged off master
• one liner for external contributors to create a new branch to open a new PR on the new history (git cherry-pick A^..B where A is the first commit and B is the last commit, from their old PR)
• validation of the new history
• archive of old history - if over 2GB, git push fails
• commits will contain their old commit id, Former-commit-id: b38c580
• rebase all branches on upstream - except ones that can't be done automatically and prebuilt branches
• deal with PRs - PRs stay open if the rest of the branches are also rewritten
• test the script at least a couple of times

Continued in #32955

[jnewb1]

20.83 MiB ❤️

[andiradulescu]

Good eye @jnewb1 - yes, it's very small ❤️

GitLab LFS is 6.3 GiB though, but that's not an issue. Since when you cloned openpilot-tiny, only 250MB of LFS files are downloaded.

I uploaded the LFS on GitLab (as you do).

One reason I keep the last lines in the script commented, is that it spams LFS and will upload LFS files everytime the reference ids of the lfs files change.

So, don't run the script until I finish it.

[adeebshihadeh]

i'm most interested in knowing the top remaining files and their sizes. does this have it?

[andiradulescu]

now you can find the latest analysis in this folder https://github.com/andiradulescu/openpilot/tree/bfg/scripts/analysis - I will try to import more files to lfs and I will keep updating the folder.

fyi external and phonelibs folder were initially deleted completely at the time of this comment #31562 (comment) - so still need to either add both folders to lfs or just add the big files inside them

[andiradulescu]

analysis of the current commaai/openpilot repo can be found here https://github.com/andiradulescu/openpilot/tree/bfg/scripts/analysis-before

[andiradulescu]

My strategy (like it or hate it):

1. push new master as “main” branch (new branch)
2. put a github action on master on every push (to master) to take the last commit that was pushed to master and cherry pick it on “main”
3. make “main” the main branch of the github repo
4. everyone works as they worked before, just that when they start a new feature they should now start from “main”
5. for some time there will be PRs opened for “main” and PRs open for “master”
6. we set a deadline for 30 days for everyone to move to “main”
7. when we reach the deadline we close all opened PRs to “master” and remove the “master” branch (and push it to archive repo)
8. clean all other branches we don’t need anymore
9. clean the repo of anything else that might need to be cleaned so that a new clone is 20MB not 3GB

The strategy takes 30 days to apply, but shouldn’t disrupt anyone in the 30 days.

After the 30 days, will be disruptive for users that need to move forcefully on main.

Didn’t do this strategy before. It’s a new idea.

[devtekve]

My strategy (like it or hate it):

1. push new master as “main” branch (new branch)
2. put a github action on master on every push (to master) to take the last commit that was pushed to master and cherry pick it on “main”
3. make “main” the main branch of the github repo
4. everyone works as they worked before, just that when they start a new feature they should now start from “main”
5. for some time there will be PRs opened for “main” and PRs open for “master”
6. we set a deadline for 30 days for everyone to move to “main”
7. when we reach the deadline we close all opened PRs to “master” and remove the “master” branch (and push it to archive repo)
8. clean all other branches we don’t need anymore
9. clean the repo of anything else that might need to be cleaned so that a new clone is 20MB not 3GB

The strategy takes 30 days to apply, but shouldn’t disrupt anyone in the 30 days.

After the 30 days, will be disruptive for users that need to move forcefully on main.

Didn’t do this strategy before. It’s a new idea.

If I can add some input, for the step 6 I’d only apply it to new PRs, otherwise it gives a 30day timeout for existing WIP which might be a bit tight. New PRs after the deadline should be rejected in favor of PRs to main

[adeebshihadeh]

RE: rollout, I'm not interested in dealing with a slow rollout, so we'll do it quickly and put some effort into rebasing PRs automatically for people.

[andiradulescu]

Small update here: rebasing master onto v0.7.1 took more than expected until I found the proper hacks. Check handle_conflicts() in my latest commit.

Right now, I'm testing this since rebasing 12040 commits is not that fast.

And during the rebase I get some logs like this:
dropping a4cec444eb3b158176f0650b414090197e7610d4 bump version to 0.8.2 -- patch contents already upstream
and investigating if this is correct or not.

[adeebshihadeh]

Anything I can help with on this? Is there a particular blocker?

[andiradulescu]

The current blocker is me. Will try to unblock myself and make more progress today.

Otherwise, you can unlock the bounty. Maybe @devtekve is interested.

[devtekve]

The current blocker is me. Will try to unblock myself and make more progress today.

Otherwise, you can unlock the bounty. Maybe @devtekve is interested.

Thanks ❤️ but not really, if it’s for me don’t unlock it! I was just chiming in. If there’s anything I can help with feel free to ping me tho

[github-actions]

Thanks for contributing to openpilot! In order for us to review your PR as quickly as possible, check the following:

• Convert your PR to a draft unless it's ready to review
• Read the contributing docs
• Before marking as "ready for review", ensure:
  • the goal is clearly stated in the description
  • all the tests are passing
  • the change is something we merge
  • include a route or your device' dongle ID if relevant

[andiradulescu]

@adeebshihadeh I edited the first comment with some checkboxes - #31562 (comment) - let me know what you think

[maxime-desroches]

Merging as is, feel free to open a pr if you need to add something to your script. We still have to wait untill all the cars stuff is out before actually doing the rewrite

[andiradulescu]

Glad seeing this merged! I’m still running it and changing small things to it, I’ll open a new PR after I stop testing and polishing.