I reported this on their HackerOne many years ago (2018 it seems) and they said it was working as intended. Conclusion: don't use private forks. Copy the repository instead.
Here is their full response from back then:
> Thanks for the submission! We have reviewed your report and validated your findings. After internally assessing the finding we have determined it is a known low risk issue. We may make this functionality more strict in the future, but don't have anything to announce now. As a result, this is not eligible for reward under the Bug Bounty program.
> GitHub stores the parent repository along with forks in a "repository network". It is a known behavior that objects from one network member are readable via other network members. Blobs and commits are stored together, while refs are stored separately for each fork. This shared storage model is what allows for pull requests between members of the same network. When a repository's visibility changes (Eg. public->private) we remove it from the network to prevent private commits/blobs from being readable via another network member.
There seems to be no such thing as a "private fork" on GitHub in 2024 [1]:
> A fork is a new repository that shares code and visibility settings with the upstream repository. All forks of public repositories are public. You cannot change the visibility of a fork.
A fork of a private repo is private. When you make the original repo public, the fork is still a private repo, but the commits can now be accessed by hash.
According to the screenshot in the documentation, though, new commits made to the fork will not be accessible by hash. So private feature branches in forks may be accessible via the upstream that was changed to public, if those branches existed at the time the upstream's visibility changed, but new feature branches made after that time won't be accessible.
OK but say a company has a private, closed source internal tool, and they want to open-source some part of it. They fork it and start working on cleaning up the history to make it publishable.
After some changes which include deleting sensitive information and proprietary code, and squashing all the history to one commit, they change the repo to public.
According to this article, any commit on either repo which was made before the 2nd repo was made public, can still be accessed on the public repo.
> After some changes which include deleting sensitive information and proprietary code, and squashing all the history to one commit, they change the repo to public.
I know this might look like a valid approach on the first glance but... it is stupid for anyone who knows how git or GitHub API works? Remote (GitHub's) reflog is not GC'd immediately, you can try to get commit hashes from events history via API, and then try to get commits from reflog.
> it is stupid for anyone who knows how git or GitHub API works?
You need to know how git works and GitHub's API. I would say I have a pretty good understanding about how (local) git works internally, but was deeply surprised about GitHub's brute-forceable short commit IDs and the existence of a public log of all reflog activity [1].
When the article said "You might think you’re protected by needing to know the commit hash. You’re not. The hash is discoverable. More on that later." I was not able to deduce what would come later. Meanwhile, data access by hash seemed like a non-issue to me – how would you compute the hash without having the data in the first place? Checking that a certain file exists in a private branch might be an information disclosure, but gi not usually problematic.
And in any case, GitHub has grown so far away from its roots as a simple git hoster that implicit expectations change as well. If I self-host my git repository, my mental model is very close to git internals. If I use GitHub's web interface to click myself a repository with complex access rights, I assume they have concepts in place to thoroughly enforce these access rights. I mean, GitHub organizations are not a git concept.
> You need to know how git works and GitHub's API.
No; just knowing how git works is enough to understand that force-pushing squashed commits or removing branches on remote will not necessarily remove the actual data on remote.
GitHub API (or just using the web UI) only makes these features more obvious. For example, you can find and check commit referenced in MR comments even if it was force-pushed away.
> was deeply surprised about GitHub's brute-forceable short commit IDs
Short commit IDs are not GitHub feature, they are git feature.
> If I use GitHub's web interface to click myself a repository with complex access rights, I assume they have concepts in place to thoroughly enforce these access rights.
Have you ever tried to make private GitHub repository public? There is a clear warning that code, logs and activity history will become public. Maybe they should include additional clause about forks there.
Dereferenced commits which haven't yet been garbage collected in a remote yet are not available to your local clones via git... I suppose there could be some obscure way to pull them from the remote if you know the hash (though I'm not actually sure), but either way (via web interface or CLI) you'd have to know the hash.
And it's completely reasonable to assume no one external to the org when it was private would have those hashes.
It sounds like github's antipattern here is retaining a log of all events which may leak these hashes, and is really not an assumption I'd expect a git user to make.
Nope, me too. The whole Repo network thing is not User facing at all. It is an internal thing at GitHub to allow easier pull requests between repo's. But it isn't a concept git knows, and it doesn't affect GitHub users at all except for this one weird thing.
Not through the GitHub interface, no. But you can copy all files in a repository and create a new repository. IIRC there's a way to retain the history via this process as well.
That’s beside the point. The article is specifically about « GitHub forks » and their shortcomings. It’s unrelated to pushing to distinct repositories not magically ´linked’ by the GH « fork feature ».
You can create a private repository on GitHub, clone it locally, add the repo being "forked" from as a separate git remote (I usually call this one "upstream" and my "fork", well, "fork"), fetch and pull from upstream, then push to fork.
Honest question. Submitting these types of bugs only to get a: "we have determined it is known low risk issue..." seems like they really don't want to pay for someone else's time and dedication in making their product safer. If they knew about this, was this disclosed somewhere? If not I don't see them playing a fair game. What's the motivation to do this if in the end they can have the final decision to award you or not? To me it looks like similar to what happens with Google Play/Apple store to decide whether or not an app can be uploaded/distributed through them.
Edit: I popped this up because to me is absolutely miserable from a big company to just say: "Thanks, but we were aware of this".
I don't think they're being underhanded exactly... they're just making a terrible decision. Quoting from the article:
> The average user views the separation of private and public repositories as a security boundary, and understandably believes that any data located in a private repository cannot be accessed by public users. Unfortunately, as we documented above, that is not always true. Whatsmore, the act of deletion implies the destruction of data. As we saw above, deleting a repository or fork does not mean your commit data is actually deleted.
"We surveyed a few (literally 3) commonly-forked public repositories from a large AI company and easily found 40 valid API keys from deleted forks."
This is how your customers get their entire cloud taken over, because you made a stupid, stupid decision and instead of fixing it when warned (repeatedly!) you instead decide to just blame the customer for not reading page 537 paragraph 3 subsection B about the counter-intuitive security footgun you've left in your product.
If you published a key, you must assume someone copied it and that deleting references to it is not sufficient. You must rotate that key now, and should check whether it was used improperly. This is pretty basic incident response.
The thing about exposing commits that were only ever in a private repo is pretty indefensible, but not garbage collecting public commits on delete shouldn't matter.
Why would anyone think that a private fork is "published"!?
This is the footgun here: The UI is telling you that nobody can see the secrets you committed to your private copy, but actually it is widely accessible.
A similar example of UI-vs-reality mismatch that I've noticed recently is the Azure Store Account "public" visibility. By default, it uses your authenticated account for RBAC access checks, so if you click around it'll say something like "you don't have browse access". This looks secure, but attempting access anonymously would have succeeded!
I had a customer recently where this happened -- they clicked through every Storage Account to "check" them, convinced themselves they were secure, meanwhile they had database backups with PII accessible to world+dog!
Putting keys in repos should not be done, full stop. Even if GitHub forks weren’t public, their _private_ repos could one day be compromised. Instead, store keys in a shared vault, .gitignore the .env and have a .env.example with empty keys.
Don't blame the end user for doing something you don't want them to do if it is more convenient to do and works without immediate consequences. Redesign it or rethink your assumptions.
The bit you quoted is referring to public forks that were deleted. That sounds like a non-issue to me, and I'm not at all surprised that
1. Public "forks" are just namespaced branches that share an underlying repo
2. They don't run the garbage collector all the time
I'd be surprised if those weren't true.
Like I said, the behavior with private forks sounds indefensible.
The OP is mixing together multiple things. Being able to access deleted public data isn't that surprising and definitely isn't a security issue as far as leaking keys is concerned (it was already public. Assume it has been cloned). Being able to access private forks is a footgun/issue. They should be garbage collecting as part of public repo creation so that unreferenced commits from private forks aren't included.
As far as I can tell, they never run the garbage collector. Code I pushed to a fork that was deleted several years ago can still be accessed through the original parent repo.
> Why would anyone think that a private fork is "published"!?
Anyone who put sensitive content in a git repo should consider published anyway. Git is a decentralized tool, as a company you cannot control the amount of git remotes that may host your code. Considering your code is only hosted as a private repo in a specific remote git server is at best naive. This is without even considering the amount of copies that are stored in dev computers.
Besides, anyone who put stuff on a third party publicly accessible infrastructure should consider it published anyway as breaches happen all the time.
If you happen to have api keys stored in a git repo, the only viable response is rotating those keys.
If the criteria for bug is "not intended", and that's solely judged by the company, then broken auth et al. suddenly become part of their product design.
I can see this argument making a bit of sense, but if they documented this 3 years after the issue was reported, they don't have a way to demonstrate that they truly already knew.
At the end it boils down to: is Github being honest and fair in answering the bug bounty reports?
If you think it is, cool.
If you don't, maybe it's not worth playing ball with Github's bug bounty process
It doesn't matter if they knew. If they don't deem it a security vulnerability --- and they have put their money where their mouth is, by documenting it as part of the platform behavior --- it's not eligible for a payout. It can be a bug, but if it's not the kind of bug the bounty program is designed to address, it's not getting paid out. The incentives you create by paying for every random non-vulnerability are really bad.
The subtext of this thread is that companies should reward any research that turns up surprising or user-hostile behavior in products. It's good to want things. But that is not the point of a security bug bounty.
> The incentives you create by paying for every random non-vulnerability are really bad.
So much this. It's pretty clear that most people commenting on this thread have never been involved in a bug bounty program on the company's side.
Bug bounty programs get a lot of reports, most of which are frankly useless and many of which are cases of intended behavior subjectively perceived as problematic. Sifting through that mess is a lot of work, and if you regularly pay out on unhelpful reports you end up with many more unhelpful reports.
This particular case definitely feels like one where the intended behavior is horribly broken, but there are absolutely many cases where "this is intended" is the only valid answer to a report.
That's true, but what's stopping a company from documenting a security issue as a known (mis)behaviour/bug? [*]
Companies can join/setup a bug bounty program, and just use it as a fig leaf for pretending to care about their own product/service's security.
Of course bug bounties can and are abused daily by people who report trivial non-issues in the hope of compensation
But in the same way, companies can also be bad actors in the way that they engage with bounties. I would usually expect big names (like Google, Apple, Github, etc.) to be trustworthy...
[*] Of course what stops companies is precisely them not being seen as trustworthy actors in the bug bounty system anymore... And for now, that's a decision that individuals have to make themselves
No large company cares even a tiny bit about the money they're spending on bug bounties. They would literally lose money trying to cheat, because it would cost them more in labor to argue with people than to pay out. In reality, the bounty teams at Google and Apple are incentivized to maximize payouts, not minimize them.
If you don't trust the company running a bounty, don't participate. There are more lucrative ways to put vulnerability research skill to use.
Github chooses to store all "Github forks" in the same repository, and allow accessing things in that repository even when they are not reachable by the refs in the namespace of one "fork". That is purely a Github decision.
They could have split forks off into new repos, but then they wouldn’t be forks, in the repository sense. It was never hard to just copy a repo instead of forking it. The UX just leads people into holding it wrong.
Shouldn't that be on the config page for the repo below the "private" button with a note saying private is not actually private if it's a fork? And ditto for delete?
No large company running a bug bounty cares one iota about stiffing you on a bounty payment. The teams running this programs are internally incentivized to maximize payouts; the payouts are evidence that the system is working. If you're denied a payment --- for a large company, at least --- there's something else going on.
The thing to keep in mind is that large-scale bug bounty programs make their own incentive weather. People game the hell out of them. If you ack and fix sev:info bugs, people submit lots more sev:info bugs, and now your security program has been reoriented around the dumbest bugs --- the opposite of what you want a bounty program to do.
In my (admittedly limited) experience, whilst payouts for bugs might be seen as a positive internally, payments for bad architecture/configuration choices are less so (perhaps as they're difficult to fix, so it's politically not expedient to raise them internally).
To provide one example I reported to a large cloud provider that their managed Kubernetes system exposed the Insecure port to the container network, meaning that anyone with access to one container automatically got cluster-admin rights. That pretty clearly seems like not a good security choice, but probably hard to fix if they were relying on that behaviour (which I'm guessing they were).
Their response was to say it was a "best practice" behaviour (no bounty applicable) and that they'd look to fix and asked me not to publicly mention it. Then they deprecated the entire product 6 months later :D
That's one example but I've seen similar behaviour multiple times for things that are more architecture choices than direct bugs, which makes me think reporting such things isn't always welcome by the program owners.
Repeating myself: this almost certainly has nothing at all to do with the money they'd have to give you (I assure you, if there's even a whiff of legitimacy to your report, the people managing the bounty would probably strongly prefer to pay you just to get you off their backs) and everything to do with the warped incentives of paying out stuff like this. People forget that the whole point of a bug bounty is that the rewarded bugs get fixed; the bounty is directing engineering effort. If it directs them to expensive work they already made a strategic decision not to do, the bounty is working against them.
You would prefer this company to have made a different strategic choice about what to spend engineering time on, and that's fine. But engineering cycles are finite, so whatever time they'd spend configuring K8s differently is time they wouldn't be spending on some other security goal, which, for all we know, was more important. Software is fathomlessly awful, after all.
Yep that's the point I was making, they don't want to pay out on architecture/configuration changes if making those would be expensive/difficult.
That doesn't mean the report isn't legitimate (being cluster-admin with no authentication is generally considered not to be a good security idea, in fact it's about as bad as it could get without putting the insecure port on the Internet), but that bug bounties aren't architected to accept that kind of issue. The challenge with this is it means that bug bounty researchers won't look for that kind of (legitimate) security issue as they get to know the programs often won't pay out.
Personally, I don't ever report bug bounties for the money reward, but so I don't get shouted at by companies when I write a blog or do a talk that covers the issues :) In this case I was a bit annoyed that they combined telling me it wasn't a bug, with asking me not to mention it publicly for 6+ months (IIRC they credited me like years after the fact).
Yes. Bug bounties are not a panacea, and were never intended to be. They have specific goals, and those goals surprise technologists working outside of the security teams that run the programs. They make a lot more sense when you remind yourself that they (a) direct engineering efforts and (b) create profound incentives; those facts together sharply constrain the problems they can be applied to.
As the article pointed out, GitHub already publicly documented this vulnerability.
My employer doesn't pay out for known security issues, especially if we have mitigating controls.
A lot of people spam us with vulnerability reports from security tools we already use. At least half of them turn out to be false positives we are already aware of. In my opinion, running a bug bounty program at all is a net negative for us. We aren't large enough to get the attention of anyone competent.
> As the article pointed out, GitHub already publicly documented this vulnerability.
I'm honestly not yet convinced that is enough here - I've fallen victim to this without realizing it - the behaviour here is so far removed from how I suspect most user's mental model of github.com works. For me none of the exposed data is sensitive, but the point remains I was totally unawares it would be retrievable like this.
If the behaviour flies so against the grain, just publishing it in a help doc is not enough I'd argue. The linked article makes the exact same argument:
> "The average user views the separation of private and public repositories as a security boundary, and understandably believes that any data located in a private repository cannot be accessed by public users. Unfortunately, as we documented above, that is not always true. Whatsmore, the act of deletion implies the destruction of data. As we saw above, deleting a repository or fork does not mean your commit data is actually deleted."
The problem with this line of argument is that the fundamental workings of git are also surprising to people, such that they routinely attempt to address mistaken hazmat commits by simple reverts. If at bottom this whole story is just that git is treacherous, well, yeah, but not news.
There's a deeper problem here, which is that making the UX on hosting sites less surprising doesn't fix the underlying problem. There is a best-practices response to commiting hazmat to a repository: revoke the hazmat, so that its disclosure no long matters. You have to do this anyways. If you can't, you should be in contact with Github directly to remove it.
Sure, but the whole point of the article is that people don't know their "private" forks aren't private. You can't get in touch with GitHub if you've never had any indication that anything's wrong.
aside: I think it's questionable to say that forking isn't a git concept. it's just a branch on a different upstream. Those two upstreams could simply be two different folders on your machine, or shared server.
I supposed the branding and UI for it could be a counter argument, but then again Github allows regular branch creation / committing / merging in their UI. Their main value add (not downplaying it—it's huge) on top of git (besides ancillary things like CI / linters) is the ability to comment on a branch's diff, i.e. a PR Review.
There's an entire custom UX flow for forking on GH that is not part of git at all. I think its very fair here to discuss "fork" in the specific sense Github uses it, as its what has lead to some of the issues discussed. There are absolutely means of providing fork functionality that don't have some of the problems we are discussing, but that's not how GH chose to build it.
1. GitHub has a nasty privacy/security hole, where commonsense expectations about the meanings of common words are violated by the system.
2. Github has publicly announced that they don't care about this part of user data security (private code), so won't pay people to know tell them what they alreay know and announced.
Github won't pay you to tell them they are wrong when everyone alreay knows.
The expectations for AWS and public repository hosting are not the same. If you leaked something to a public GitHub repo you should assume that it has been cloned the second you pushed it.
For both sides it turns into a net negative. Better to keep your bugs and use them when needed or sell them to others to use if possible.
Lets get back to what we had before when multiple people can find the same bug and exploit if needed. Now we have the one person who finds the bug it gets patched and they don't get paid.
Security disclosures are like telling someone they have a spot on their face. It's not always welcome, and there's no obligation on anyone to do so, nor anyone to return the favor.
In this case, the spot turned out to be a freckle, which everyone involved already knew was a freckle (since it was documented), and if anyone owes anyone anything, it's the researcher that owes github for wasting their time.
Disagree. This is obviously a deliberate design choice with obvious implications. Expecting a bounty for reporting this is unreasonable. These kind of beg bounties are exactly what gives security "researchers" a bad name.
The security implications are also minor. The only problem really is with making a fork of a private repo public - that should only make what exists in that fork public and not any other objects. Something that was already public staying public even when you delete it from your repo is not a security issue at all. Keys you have ever been pushed to a public repo should be revoked no matter what, with or without this GitGub feature.
I wasn't really expecting a bounty, more so hoping they'd fix the issue. For example, to this day I keep having to tell people to never fork the Unreal Engine repository, instead making a manual copy, just in case.
This causes lots of problems for repositories that are private with the expectation that companies will make private forks with their own private changes.
Someone once pushed a bunch of console SDKs (under strict NDA) to a private fork without knowing this. Now that code is just there, if you can guess the commit hash, forever. Literally nothing can be done to remove it. Great.
I reported a variant of this issue that (to me) was unexpected:
* You add someone to your private repo.
* After some time, you revoke their access.
As long as they keep a fork (which you can't control) they can use this same method to access new commits on the repo and commits from other private forks.
Back in 2018, this was a resolved as won't fix, but it also wasn't documented.
For moral reasons, historically I never wrote POCs or threatened disclosure.
For companies like Microsoft, which a CSRB audit showed that their security culture 'inadequate', the risk of disclosure with a POC is about the only tool we have to enforce their side of the Shared Responsibility Model.
Even the largest IT spender in the world, the US government has moved more from the carrot to the stick model. If they have to do it so do we.
Unfortunately as publishing a 'bad practices' list by us doesn't invoke the risk of EULA busting gross negligence claims, responsible disclosure is one of the few tools we have.
It's not just GitHub and it's not just because they don't want to pay bug hunters. In my career, I have escalated multiple bugs to my employer(s) in which the response was 'working as intended'. And they wouldn't have to pay me another cent if they acknowledged the issue.
In my experience, there was two reasons for this behavior:
1. They don't want to spin dev cycles on something that isn't directly related to revenue (e.g. security)
2. Developers don't have the same mindset as someone who's whole job is security. So they think something is fine when it's really not.
I didn't find anything mentioning it online at the time. But there wasn't much time and dedication involved either, to be fair. I discovered it completely on accident when I combined a commit hash from my local client with the wrong repository url and it ended up working.
The issue had been reported at least twice and was clearly documented. GitHub knew about this and had known for years. Their replies to the two notifications were even very similar.
GitHub clearly knew. Would you prefer that a vendor lie?
companies vary wildly in their honesty and cooperation with bug bounties and develop reputations as a result. if they have a shit reputation, people stop doing free work for them and instead focus on more honest companies
there are endless examples of reports that, by the judgement of the community, are entirely valid and unknown security concerns and fit within the guidelines of a company's bug bounty program, and do not ever result in payment
I reported a different security issue to github, and they responded the same (although they ultimately ended up fixing it when I told them I was going to blog about the "intended behavior").
Imho there is an issue with the word "delete". Apparently for anyone who is hosting someone else's (private and/or sensitive and/or worthy) data is to hide it from view, but keep it around "just in case" or "because we can" or "what are you gonna do about it"?
I 'love' it when I see the words "hide", "archive", "remove", and other newspeak to avoid using the word "delete", since 'they' never actually delete (plus there are 1-2-5-10-forever years' of backups where your 'deleted' info can be retrieved relatively easy).
What does "private fork" mean in this context? I created a fork of a project by cloning it to my own machine and set origin to an empty private repository on GitHub. I manually merge upstream changes on my machine.
It’s not. The feature here works because a network of forks known by GitHub has a unified storage, that’s what makes things like PRs work transparently and keep working if you delete the fork (kinda, it closes the PR but the contents don’t change).
the issue is the `fork` mechanism of github is not semantically like a `git clone`
it's more like creating a larger git repo in which all forks weather private or not are contained and which doesn't properly implement access management (at least point 2&3 wouldn't be an issue if they did)
there are also some implications form point 1 that forks do in some way infer with gc-ing orphan commits (e.g. the non synced commits in he deleted repo in point 1) at least that should be a bug IMHO one which also costs them storage
(also to be clear for me 2&3 are security vulnerabilities no matter if they are classified as intended behavior)
Because you never git pushed to the fork it's not aware of your repo, you're ok.
What I don't know is if in 3 months you DO set your remote origin to that fork to for instance, pull upstream patches into your private repo, you're still not pushing, only pulling, so I would THINK they'd still never get your changes, but I don't know if git does some sort of log sync when you do a pull as well.
Maybe that would wind up having the commit hash available.
It would not even be that hard to fix it; private forks should always just be automatically copied on first write. You might lose your little link to the original repo, but that's not as bad as unintentionally exposing all your future content.
> After some internal discussion, we have determined this is a known low risk issue. We may make this functionality more strict in the future, but don't have anything to announce now. As a result, this is not eligible for reward under the Bug Bounty program. Below is a reference to our instructions for users to remove sensitive data from a repository. https://help.github.com/articles/removing-sensitive-data-fro...
To be fair, in the true git sense, if a "fork" is really just a branch, deleting the original completely would also mean deleting every branch (fork) completely
Users should never be expected to know these gotchas for a feature called "private", documented or not. It's disappointing to see GitHub calling it a feature instead of a bug, to me it just shows a complete lack of care about security. Privacy features should _always_ have a strict, safe default.
In the meantime I'll be calling "private" repos "unlisted", seems more appropriate
No, it really isn’t. Anyone who uses that word that way is just factually incorrect, and probably pretty irresponsible depending on the context. Software should not tell lies.
I think you are referring to tombstoning. That's usually a temporary process that may immediately delete the underlying data, keeping a tombstone to ensure the deletion propagates to all storage nodes. A compaction process purges the underlying data (if still present) and the tombstones after a suitable delay. It's a fancy delete that takes some time to process, but the data is eventually gone. You could turn off the compaction, if you wanted.
I believe Kafka make deletion difficult, since it's an append-only log, but Kafka doesn't work well with laws that require deletion of data, so I don't believe it's a popular choice any longer (I.E. isn't modern).
If you run a DELETE FROM in any modern sql engine, which is the absolute best you could expect when asking for a delete in the UI^, the data is nowhere near gone. It’s still in all the backups, all the WALs, all the transactions that started before yours, etc. It’s marked for eventual removal, and that’s it. Just as the definition of delete I provided says.
^ (more likely they’ll just update the table to set a deleted flag)
To me, the idea that the deletion takes time to complete doesn't negate the idea that the data will be gone once the process completes.
WAL archive and backups are external systems. You could argue that nothing supports deletion because an external backup could exist, but that's not a useful conversation.
Going back to the point of the the thread, we agree the deleted data is not erased. The user is unable to access it through normal mechanisms, but the existence of side channels that could reveal it does not negate the idea that it has truly been “deleted”, especially when one looks at the historical context surrounding that word.
Can you point to an example of a modern database that "supports deletion" but keeps the data around forever? Maybe I've just used different tools than you. Knowing modern data retention concerns I'd be surprised if such a thing existed.
Who said anything about that? We’re talking about side channels and eventual^TM deletion. Given enough time no information will remain anywhere, sure. But that’s not very relevant.
I think we are trying to define the word "delete". You found an archaic definition and are trying to use it in a modern technical setting. You've claimed that modern databases delete without actually removing data but haven't pointed to which systems you are talking about. I'm familiar with tombstoning, either as a "soft-delete" or as part of an eventual deletion process. But I've never seen that called deletion as that would be very confusing.
Pointing to which database you are talking about should clear this up quickly.
I don't think it's reasonable to talk about backups here. A backup is external to the database so it inherently cannot delete it. Similar to how a piece of paper cannot destroy a photograph of the paper, but burning the paper destroys it.
I used the first definition of delete I found which, while arguably “archaic”, matches the modern technical term almost exactly. We’d typically call that a well known word with a clear meaning.
And sure, the DELETE FROM statement in postgres - or any other standards compliment sql db I know.
In technical writing you often don't want to use the dictionary for definitions, similar to how words in a contract can have unexpected meaning in a legal setting.
For Postgres you've got to consider vacuum. Auto vacuum is enabled by default. Deleted rows are removed unless you go out of your way to make it do something different.
Imagine the data that was deleted is of the highest level of illegality you can imagine. Under no circumstance can your service be associated with that content.
- What was your "definition of delete" again?
- You mentioned some of the convenient technical defaults your frameworks and tools provide out-of-the-box, can you think of ways to improve the situation?
(You might re-run delete requests after restoring a backup; transaction should resolve in a timely fashion, failed deletes can be communicated to the user quickly etc.)
We are missing the point here. The GP was claiming that delete meant something other than adding a mark to an item that you want to eventually be removed from the system. It doesn’t.
I understand that you describe the status quo in many systems today.
However, besides the technical aspect you talked about the "absolute best you could expect when asking for a delete in the UI^".
I think this where I, other posters in the thread, most people, and probably the GDPR and other legislature, would disagree. We expect significantly more effort to clean up deleted data.
This includes, for example, the ability to delete datasets from backups, as well as a general accountability of how often and where all the data is stored and if, and when a deletion process is complete.
Nope. GDPR allows deleted data to be retained in backups so long as there is an expiration process in place. Doesn’t matter how long it is. But certainly nobody has a right to forcing a company to pull all of their backups from cold storage and trove through them all any time any deletion request takes place. That’d be the quickest path to Distributed Denial of Bank Account Funds imaginable. Even the GDPR isn’t that bone-headed.
But yes, it is part of the law that the provider should tell you that your data isn’t actually being erased and instead it will be kept around until they get around to erasing everything as part of their standard timelines. But that knowledge doesn’t do anyone much good.
> CNIL confirmed that you’ll have one month to answer to a removal request, and that you don’t need to delete a backup set in order to remove an individual from it.
But GitHub is keeping this stuff indefinitely. No long expiration, no probability of eventual disk overwriting, nothing. All they're doing is shutting the front door without shutting the side door.
Interesting point about the GDPR; I will soften my point to mean that lawmakers have started (late) to regulate data retention / deletion and the rights of users in general and that might be a trend for the future.
However I would like to avoid the impression that with the description of the technical status quo the topic is settled. To do so I would go back to my previous point: Imagine some truly illegal pictures are in that cold storage backup, and one day you might have to restore that data. (Since aparently the user's wish to delete data is not quite as respected as certain other hard legal requirements regarding content)
What solutions to mitigate the situation could a company, or backup tool/web framework etc. reasonably come up with? Maybe check the restored data against a list of hashes/IDs of to-be-deleted-data?
Every modern file system works like this too. Then there’s copy-on-write snapshotting and SSD wear leveling to worry about. Data isn’t actually destroyed until the space is reused to store something else at an indeterminate point in the future.
Or when its encryption key is overwritten.
But it probably is a good idea to stop returning deleted data from web APIs.
this is why when I'm building confirm UI, I prefer the term "destroy?" on the confirm action. It's much clearer to the user that this is a destructive and irreversible action and we will be removing this data/state.
No, deleted is a word for deleted. But we started saying things were "deleted", while our eyes flicked to the stack of backup tapes in the corner, acknowledging the white lie, because really deleting things conflicted with other priorities and was hard. And we left it there, until privacy regulations came along and it turned out not using the normal definition of deleted could get you sued. So IMO Github is wide open to paying damages to the first person able to demonstrate them.
For the benefit of anybody thinking "with gitlab I'm safe from this": If you're saying (and perhaps you're not) that some other git hosting service
- gives you control over gc-ing their hosted remote?
- does not to your knowledge have a third-party public reflog or an events API or brute-forceable short hashes?
if so, especially the second of those seems a fragile assumption, because this is "just" the way git works (I'm not saying the consequences aren't easy to mentally gloss over). Even if gitlab lacks those things curently (but I think for example it does support short hashes), it's easy to imagine them showing up somehow retroactively.
If you're just agreeing with the grandparent post that github's naming ("private") is misleading or that the fork feature encourages this mistake: agreed.
Curious to know if any git hosting service does support gc-ing under user control.
1. Create a private repo R
2. Create a private fork F of R
3. Push commits to the fork F
4. Make R public
The commits pushed to F prior to R being made public will become de facto public, even though F has always been a private fork. The post makes clear that commits pushed to F after R is made public are placed into a separate, private fork network.
So basically, if you ever intend to open source anything, never do it to an existing private repo. Always start a from-scratch repo to be the root of your new public project.
> if I want to host something private, I will choose another vendor.
Or you know, self-host, preferrably on-prem.
Basic git hosting only needs a sshd running on the server. If you want collaborative features with a web UI then there are solutions for that available too.
I've been happy with Jetbrains Space (now Space Code); but I'm using it for private, professional work and paying for it, so perhaps that isn't what you mean.
That might be a bit too strict. I'd still expect my private repos (no forks involved) to be private, unless we discover another footnote in GH's docs in a few years ¯\_(ツ)_/¯
But I'll forget about using forks except for publicly contributing to public repos.
> Users should never be expected to know these gotchas for a feature called "private".
Yes, the principle of least astonishment[0] should apply to security as well.
Specifically about the feature called "private", the only gotcha seems to be that when the upstream transitions from private to public, it may unexpectedly take more data public than desired, right? The other discussed gotchas were all about deleting public data not actually being deleted or inaccessible.
Disagree. If you're using a service, understand how it works.
Not everything needs to be designed for idiots and lazy people, it's ok for some tools and services, especially those aimed at technical people and engineers to require reading to use properly and to be surprising or unintuitive at first glance.
There's got to be a word for these kinds of ridiculous arguments which use personal responsibility as a cudgel against a systematic fix.
I agree generally that interfaces have been dumbing down too far, but "private is actually not private and it's on you for not knowing that, idiot B)" is a weird place to be planting that flag.
There should probably also be a word for the belief that when a system doesn't work how you want it to, that is so obviously a systematic problem that needs fixing rather than, for example, evidence of differing goals or priorities that it is reasonable to describe anyone who thinks otherwise as ridiculous.
The biggest gotcha here is probably that if you start of with a private repo and a private fork, making the repo public also makes the fork "public".
GitHub may very well say that this is working as intended, but if it truly is then you should be forced to make both the repo and fork public at the same time.
Essentially "Making repo R public will make the following forks public as well 'My Fork', 'Super secret fork', 'Fork that I deleted because it contained the password to my neighbours wifi :P'.
OK. I'm not sure if the last one would actually be public, but I wouldn't be surprised if that was "Working as intended(TM)" - GitHub SecOps
Any time you make a private repo public it’s best to just copy that code into a new public repo and leave the private repo private. Otherwise have to audit every previous commit and every commit on every fork of your private code.
I agree. The other cases may be mildly surprising, but ultimately fall firmly into the category of "once public on the internet, always public." Deleting a repo or fork or commit doesn't revoke an access key that was accidentally committed, and an access key being public for even a microsecond should be assumed to have been scraped and usable by a malicious actor.
If you have a private repo, you would assume that nothing in that private repo becomes public unless you do something very explicit.
The issue here is that if you have a private repo and a private fork of that repo. If you make the private repo public and keep the fork private, then you are not explicitly told that your fork is actually public, whether you want to or not.
it's a bit of that you have to know the sha, and that's quite unique. it's apparently unique enough for Google photos to "private" share without logins
For the first two, git is based on content addressable storage, so it makes sense that anything that is ever public will never disappear.
I can sympathize with someone who gets bit by it, as it might not have occurred to them, but it’s part of the model.
The third strikes me as counter-intuitive and hard to reason about.
P.S. If you publish your keys or access tokens for well known services to GitHub and you are prominent enough, they will be found and exploited in minutes. The idea that deleting the repository is a security measure is not really worth taking seriously.
> For the first two, git is based on content addressable storage, so it makes sense that anything that is every public will never disappear.
this isn't quite right
content addressable storage is just a mean of access it does
- not imply content cannot be deleted
- not imply content cannot be access managed
you could apply this to a git repo itself (like making some branches private and some not) but more important forks are not git ops, they are more high level github ops and could very well have appropriate measurements to make sure this cannot happen
e.g. if github had implemented forks like a `git clone` _non of this vulnerabilities would have been a thing_
similar implemented different access rights for different subsets of fork networks (or even the same git repo) technically isn't a problem either (not trivial but quite doable)
and I mean commits made to private repositories being public is always a security vulnerability no matter how much github claims it's intended
You're right that I shouldn't have given the impression that content addressed storage means as a technical matter that public content must never disappear. The phrasing was a bit sloppy. GitHub could, as a technical matter, choose to hide content that had previously been made public.
Nonetheless, given that GitHub exists to facilitate both anonymously pulling the entire history of the repository, and given that any forks would contain the full contents of that repository, it is very natural that GitHub would take the "once public always public" line.
> and I mean commits made to private repositories being public is always a security vulnerability no matter how much github claims it's intended
I specifically said the third use case was different, because it is the one that doesn't involve you explicitly choosing to publish the commits that contain your private information. I did not and would not defend GitHub on that point.
I agree the 3rd is by far the worst of the offenders. But even the first two should have more visibility. For example, by notifying users during deletion of forked repos that data will still be available.
The exact UX here is debatable, but I don't think security warnings buried in the docs is enough. They should be accounting for likely misunderstandings of the model.
Even if it wasn't forked, it could be cloned. Should that be part of the warning?
I wouldn't mind a disclaimer when you delete a repository that any information that repository ever contained is likely to have already been downloaded and stored. Per the comment I added, I'm not sure it would really help that much, but it would not be harmful.
It couldn't hurt, but that isn't the misunderstanding I'm worried about.
As described in the first example of the article, you can make a fork, commit to it, delete your entire fork, and yet the data will still be accessible via the parent repo, even though no one ever forked or cloned or saw your fork. That is not intuitive at all.
You can say "Well just consider any data that has ever been public compromised forever", and indeed you should, but this behavior is still surprising and could bite devs even if they know they should follow the advice in that quote.
Consider a situation like this...
Dev forks, accidentally pushes a secret or some proprietary code in a commit, and immediately deletes the fork. They figure it was only up for a very short time, now it's gone, risk someone saw it is low. They don't bother rotating, because that would be a major operational pain (and yes, it shouldn't be, but for many orgs it is).
Is this dev making a mistake? Of course. That's not good security thinking. But their assessment of the risk being low might actually be correct if their very reasonable mental model of deletion were correct. But the unintuitive way GH works means that the actual risk is much higher than their reasoning led them to believe.
> As described in the first example of the article, you can make a fork, commit to it, delete your entire fork, and yet the data will still be accessible via the parent repo, even though no one ever forked or cloned or saw your fork. That is not intuitive at all.
But isn't that only the third vulnerability, that private forks are implicitly made public?
> For the first two, git is based on content addressable storage, so it makes sense that anything that is ever public will never disappear.
No one can, with a straight face, say that they don’t restrict access because “this is just how the technology works”. Doesn’t matter if it is content addressable or an append-only FS or whatever else.
Even for some technology where the data lives forever somewhere (it doesn’t according to Git; GitHub has a system which keeps non-transitively referenced commits from being garbage collected), the non-crazy thing is to put access policy logic behind the raw storage fetch.
> git is based on content addressable storage, so it makes sense that anything that is every public will never disappear.
No. That doesn't make sense. It only sounds vaguely plausible at first because content addressable storage often means a distributed system where hosting nodes are controlled by multiple parties. That's not the case here, we're only talking about one host.
Imagine we were talking about a (hypothetical) NetFlix CDN where it's content addressed rather than by UUID. Would anyone say "they forgot to check auth tokens for Frozen for one day, therefore it makes sense that everyone can watch it for free forever"?
Since Netflix neither allows anonymous users to fully download Frozen without DRM, nor allows authorized users to upload derivative works that are then redistributed to the public, I think there may be some relevant differences here.
They do remove content when their licence expires, though. So imagine instead Netflix allowing users to find and watch expired series by hash, then telling the copyright owners they can't fully delete the series because something something content-addressing.
> I've used github for a long time, would not have expected these results, and was unnerved by them.
So you've used it heavily, but haven't read the docs or thought about how forks work, and are now surprised. This seems like a learning opportunity, read the docs for stuff you use heavily, read the man pages and info pages for tools you rely on.
None of this seemed surprising to me, perhaps because I've made PRs, seen that PRs from deleted repositories are still visible, and generally have this mental model of "a repository fork is part of a network of forks, which is a shared collection of git objects".
Congratulations, you developed the right intuition.
However in UX/DX the question isn't whether users can develop the right intuition based on how they interact with software over time and reading through the documentation but how to shorten the time and effort necessary for that, ideally so that a single glance is enough.
Do you think reading all the documentation for every feature of every tool you use in your life is a good use of your time and something that should be expected of everyone? As someone developing software used by other people, I don't.
IMO, the real vulnerability here is the way the Github Events archive exposes the SHA1 hashes of the vulnerable repositories. It would be easy to trawl the entire network to access these deleted/private repositories, but only because they have a list of them.
Similar (but less concerning) is the ability to use short SHA1 hashes. You'd have to either be targeting a particular repository (for example, one for which a malicious actor can expect users to follow the tutorial and commit API keys or other private data) or be targeting a particular individual with a public repository who you suspect might have linked private repositories. It's not free to guess something like "07f01e", but not hard either.
If these links still worked exactly the same, but (1) you had to guess 07f01e8337c1073d2c45bb12d688170fcd44c637 and (2) there was no events API with which to look up that value, this would be much, much less impactful.
First step would be to have them acknowledge a documented behavior which was part of their original design 16 years ago, is something that needs to be fixed.
As someone who has used git and GitHub extensively over that time, none of what the author documented was a surprise to me.
However, I also remember when people were trained to do a "Save As" when preparing a final Word document or Powerpoint for sharing with a third party. That certainly bit enough business users that Microsoft eventually changed the default behavior.
It's not doing Save As that bit people. Think of a .doc file as a bad database format. It gets lots of in-place overwrites, and fragments of old versions stick around.
Right, OLE documents are comparable to read/write filesystems inside a file. The only mechanism given to make sure it is sparse was to create a new filesystem by having the application walk the existing one, basically a copy-based garbage collection.
Powerpoint files can be megabytes larger due to unused graphic artifacts; Word documents may contain older revisions of the text that contain deleted sections. Other things like the MSI installer file format are also OLE documents.
Microsoft eventually made Save in Office apps always create a new sparse filesystem to prevent these problems.
If people are surprised by this - and clearly a non-trivial number of people are - then even if the behavior works as intended it should be indicated in the UI at critical points.
GitHub is not simply a UI to actual git repositories it hosts. It also carries a lot of data that is not stored in the repository itself. The UI deliberately blends the two types of interactions. There's no such thing as "creating a pull request" in git, for example.
It's not at all unreasonable for a user to assume "forking" merely creates a copy with an upstream origin. Read through the steps again:
1. You fork a public repository
2. You commit code to your fork
3. You delete your fork
Note that from a user's point of view, they only committed code to their "copy" of the repository, i.e. their own repository. They never pushed it upstream or created a pull request that references it. That abstraction is clearly wrong if you look at what actually happens but it's not difficult to see why a user might think this way, especially given that "a fork" often simply means "a copy", i.e. something standalone that then goes on to diverge from its origins (e.g. "Edge is a fork of Chromium" or "MariaDB is a fork of MySQL").
Of course the mistake is that the fork is not a copy. The fork isn't a fork (i.e. a separate copy that shares the original's history), it's a view of the original repository with its own refs. The commits are added to the same repository, only the refs that reference them aren't. This makes sense architecturally but it means most metaphors and analogies people likely bring to the table break down because they assume a fork is a copy, not a collection of refs layered over a shared repository of commits - after all "allowing stranges to add commits to the repository" is what PRs are for. Except of course that's not what PRs do then, PRs actually allow strangers to add references to commits to your branches.
We all need to embrace: Nobody has ever been impressed that you already knew something. When people share a discovery with you, it’s not about you. It’s about them and their joy of discovery. They want to share that joy with you.
If they're sharing the joy of discovery that's great. Lots of people though are gloating. "Haha, you're stupid, you didn't know X and I did". In other words, they're the ones not being charitable by assuming you don't already know X.
Trying to think of an example it usually goes something like
A: We should do X
B: No. See this document (the sharing part)
A: I wrote that document and I'm telling you we should do X (the "I already knew this" part)
Hubber here (same username on github.com). We in GitHub's OSPO have been working on an open source GitHub App to address the use case where organizations want to keep a private mirror of an upstream public fork so they can review code and remove IP/secrets/keys that get committed and squash history before any of those changes are made public. Getting a beta release this week, in fact - check it out, I'm curious what yall think about the approach
Looks like a promising tool and workflow to mitigate the risks we are discussing here. If you haven’t already done so, it might help the discussion here if you could highlight how this app deals with the issues outlined. Is the intent of the mirror repo creation that it’s more-or-less equivalent to “git clone —mirror”? I took a quick look at the code, and didn’t see a direct correspondence with “git clone —mirror” when creating the mirror repository.
As it pertains to the post, since that private mirror is disconnected, none of the concerns about accessing deleted data apply.
The downside is that you don't get any of GitHub's performance and UI affordances from the fork network. But for the use case of private iterations on work headed for a public upstream, that's a trade-off that seems worth making.
Can this be used to host illegal content? I.e.: fork a popular repo, commit a pirated book to the fork, delete the fork, use the original repo to access the pirated book?
What would github do after receiving a DMCA request in that case?
I've seen bots make that kind of PR spam a few times. They'll make a PR that adds a random HTML or markdown file or whatever containing gambling spam or whatever and then presumably post links to github.com/$yourorg/$yourrepo/blob/$sha/thatfile I can't link an example because all the ones I know about were nuked by GH Support.
It can be used to make it look like another project posted the content (though there is a warning: "This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.").
You can't host anything this way that you can't already host in your own repository, and GitHub does have a way to remove content that will make it inaccessible, whether in your repository or through another.
There's quite a long list of "open core" companies whose models are, start from a private repository (i.e. company is in stealth), make a private fork that will include for-profit code with enterprise features, make the original repository public so that the core will be open-source.
That GitHub is telling these companies, and bear in mind that these companies are paying customers of GitHub, yeah we don't care that your private proprietary code can be hacked off GitHub by anybody, is incredibly disturbing. Is there really not enough pressure from paying customers to fix this? Is Microsoft just too big to care?
In response to the end of the article "it’s important to note that some of these issues exist on other version control system products." I actually have experience helping someone with an issue on BitBucket with PII data that you can't rotate.
Once we eliminated the references in the tree and all forks (they were all private thankfully), we reached out to BitBucket support, and they were able to garbage collect those commits, and purge them to the point where even knowing the git hashes they were not locatable directly.
> The implication here is that any code committed to a public repository may be accessible forever
That's exactly how you should treat anything made available to the public (and there's no need for the subsequent qualifier that appears in the article—"as long as there is at least one fork of that repository").
Sometimes I wonder if all the security features GitHub slathers on top of `git` lull people into a false sense of security when fundamentally they're working in a fully distributed version control system with no centralized authority. If your key is leaked the solution is to invalidate the key not just synthetically alter your version of history to pretend it never happened.
Unless you specifically know and understand the ramifications of this GitHub idiosyncrasy, you have no way to tell that your key was possibly leaked. GitHub never informs you that someone accessed a commit created in your private fork.
Most of this report is just noise. GitHub repos are public. Public stuff can be shared. Public stuff shared previously and then deleted is "still available", but it was shared previously and not really subject to security analysis.
The one thing they seem to be able to show is that commits in private branches show up in the parent repository if you know the SHAs. And that seems like a real vulnerability. But AFAICT it also requires that you know the commit IDs, which is not something you can get via brute forcing the API. You'd have to combine this with a secondary hole (like the ability to generate a git log, or exploiting a tool that lists its commit via ID in its own metadata, etc...).
Not nothing, but not "anyone can access private data on GitHub" as advertised.
> it also requires that you know the commit IDs, which is not something you can get via brute forcing the API
Well, GitHub accepts abbreviations down to as short as four hex digits... as long as there's no collision with another commit, that's certainly feasible. Even if there is collision, once you have the first four characters you can just do a breadth-first search
There's a whole section here about how to brute force the hashs. You don't even need the full hash... just a shortened version using the first few chars.
I'm dubious. Searching for globally unique commit IDs is still a least a million+ request operation. That's easy enough in a cryptographic sense but the attack in question requires banging a web UI, which is 100% for sure going to hit some abuse detector. I really don't think you can do this in practice, and the article certainly doesn't demonstrate it.
I maintain a pretty popular template for SaaS websites. Every few weeks someone would send a PR with all their private fork data, then quickly try to delete it.
Making it a "template" repo mostly fixed the issue. That creates a copy instead of a fork. However it still happens from time to time.
I think the first two points are a result of private data (commit/fork/issue) being able to refer to public data without making the reference public.
Say a private commit depends on a public commit C. Suppose in the public repo, the branch containing C gets deleted and C is no longer reachable from the root. From the public repo's point-of-view, C can be garbage-collected, but GitHub must keep it alive, otherwise the deletion will break the private commit.
It would be "a spooky action at a distance" from the private repo's POV. Since the data was at a time public, the private repo could have just backed up everything. In fact, if that's the case, everyone should always backup everything. GitHub retaining the commit achieves the same effect.
The public repo's owner can't prevent this breakage even if they want to, because there's no way to know the existence of this dependency.
The security issue discussed in the post is a different scenario, where the public repo's owner wants to break the dependency (making the commit no longer accessible). That would put too much of a risk for anyone to depend on any public code.
My mental model is that all commits ever submitted to GitHub will live forever and if it's public at one time, then it will always be publicly accessible via its commit hash.
Really the only semi-interesting part of this is "if you make a private repo public, data from other private forks might be discoverable", but even that seems pretty minor, and the best practice for taking private repos public is to copy the data into a new repo anyway.
Is that a best practice in hindsight, or because it was known to some, that this issue exists, or for what other reason do you consider it a best practice? Git history?
When making a private repo public, there's a high chance that there was stuff in the private repo that isn't necessarily ok to make public. It's a lot easier to just create a new public repo containing all the data you want to make public than it is to reliably scrub a private repo of any data that shouldn't be there.
More generally, you probably want to construct a new history for the public repo anyway, so you'll want a brand new repo to ensure none of the scrubbed history is accessible.
I worked in Professional Services at AWS for a little over three years. There was a fairly easy approval process to put our work out on the public AWS Samples (https://github.com/aws-samples) repository once we removed the private confidential part of the implementation.
I always started a new repository without git history. I can’t imagine trying to audit every single commit.
I found the post you're replying to helpful (and it made me laugh): I've come across the abbreviation POLA many times, with its non-jokey meaning "principle of least authority". I've also come across "principle of least astonishment" (Larry Wall or some other Perl contributor maybe?) but I'd never noticed that was (presumably?) a jokey reference to principle of least authority - I guess because I came across the joke first back was I was barely a programmer and I've never seen it abbreviated.
But maybe it never was a reference to POLA proper - "principle of least privilege" is more widespread I think, outside of the object capability community. And maybe "least astonishment" came first!
Unrelated, but another interesting one is any non-admin contributors being able to add (and I believe update) secrets in a private repo for use in GH actions. It can’t be done via the UI, but can be done via the API or VSCode extension.
When I looked into it a while back, apparently it is intended behavior, which just seems odd.
You're saying there's a github API that takes as an argument a secret, and creates a git commit containing that secret? I'm very surprised. Can you provide a reference to the API call?
Here is their full response from back then:
> Thanks for the submission! We have reviewed your report and validated your findings. After internally assessing the finding we have determined it is a known low risk issue. We may make this functionality more strict in the future, but don't have anything to announce now. As a result, this is not eligible for reward under the Bug Bounty program.
> GitHub stores the parent repository along with forks in a "repository network". It is a known behavior that objects from one network member are readable via other network members. Blobs and commits are stored together, while refs are stored separately for each fork. This shared storage model is what allows for pull requests between members of the same network. When a repository's visibility changes (Eg. public->private) we remove it from the network to prevent private commits/blobs from being readable via another network member.
reply