Asciidoctor: support include directives for other asciidoc files

[miltador]

A lot of time has passed since latest comments from devs in #172 and #335.
I think after years things have changed in your infrastructure, why not to bring up this issue again with some more input and news?

For example, AFAIK GitHub started to use containers which could help to isolate things from unintentional access and improve security.

Also take a look at asciidoctor/asciidoctor#1088. The author of Asciidoctor is ready for a conversation about this, there is even proposed a way to implement this with custom include preprocessor. If there are still some concerns about resolving the issue, please provide a constructive feedback so both sides can collaborate.

[chevdor]

I totally 👍 this request. Includes are an amazing feature of Asciidoc over Markdown and would help a lot providing users with good and up-to-date documentation.

I would add image to the same bucket. I think it is not supported today, probably for the same reason includes are not.

[mojavelinux]

I would add image to the same bucket. I think it is not supported today, probably for the same reason includes are not.

The image macros are supported on GitHub. You can see them in action here: https://github.com/asciidoctor/atom-asciidoc-preview

[mojavelinux]

If GitHub is committed to enabling includes for AsciiDoc files, I'm willing to do whatever needs to be done in Asciidoctor to make it happen. I've already added an extension point so the include directive can be handled by a custom function. That allows the GitHub code to resolve the file from the git repository instead of from the file system.

[ericis]

Fully support this. Seems like a basic capability and we're struggling without it 👎
http:https://asciidoctor.org/docs/user-manual/#include-directive

[nailgilaziev]

it will be great to use a full power of asciidoc from github frontend.
For example I use restdocs & spring to generate snippets for documentation of my API.
Most of documentations is handwritten and only generated snippets was included.
I want to store this documentation in github and collaboratively edit it with others via github instruments(pull requests).

[kavaliro]

This issue was an unpleasant surprise. I agree with @ericis; this is fundamental functionality which is expected by Asciidoc users.

Since @mojavelinux has enabled custom handling , this seems simple:

1. check for infinite recursion
2. only allow relative includes. (i.e., only include files within the repository.)

This seems fairly solvable. Maybe I'm not seeing the full extent of it?

[denuno]

I think it's been a solvable problem for a while now, technically-- at this point it's just someone flipping a couple switches by the looks of it.

This is hands down one of the best features of AsciiDoc and it's a crying shame that there is so little support for it in git-related tooling (GH, Gollum, GitBookIO, etc.).

FWIW, it's not like images are that much safer than other types of content, but for years now I could generate images of my source code and include it that way (ridiculous, as no copy/paste), or generate the content itself and include it that way (which defeats the purpose of nice rendering via GH out of the box).

Someday, it would be nice to move into the brave new future and get some include::sourceFile[tags=tagName] type stuff going... I know not many people like writing docs but this stuff is gold for folks who-- well, may not love writing them, but at least who want to write them as smartly as possible-- and right now those folks have to do the work twice to get it to look nice, basically, and while it can be automated... it shouldn't need to be, in this day and age! 😃

[obilodeau]

So it's pretty clear the reason it hasn't been implemented yet is security. I believe it is also a reason why GitHub's staff doesn't really engage with the community in order to solve that problem because it would require them to expose their threat model and pieces of their architecture which could help attackers.

FWIW, it's not like images are that much safer than other types of content, but for years now I could generate images of my source code and include it that way (ridiculous, as no copy/paste), or generate the content itself and include it that way (which defeats the purpose of nice rendering via GH out of the box).

You are missing the broader point. Include processing is more complex, not because the destination is dangerous, but because of the way it interacts. Just as an example: An include can include includes which could include the first top-level include. This would DoS a naive parser. Analogous to the Billion Laugh Attack and it could be done in so many ways that it's hard to prevent them all with parser restrictions. Another different, yet complex to fix, example: AsciiDoc includes can be URIs, URI includes could be used to perform SSRF attacks in order to explore GitHub's infrastructure in means that shouldn't be allowed.

So I think the reason why GitHub is not doing it is because <1% of their users care and there are numerous ways to abuse the feature so doing it safely is hard. Implementing a hard feature for <1% is not worth the resources (or the risk).

Don't shoot the messenger here: I want AsciiDoc includes to work on GitHub! It's just that I can see why it's a tricky one to implement.

[denuno]

Nothing wrong with sharing how you think-- no shots here, but I disagree that it's security holding this up per se. 😃

Mainly because in the comments above, both injection attacks, and recursion have solutions (and have been solved elsewhere in github infrastructure already)... it's weird that infinite recursion is even on the list, IMHO-- same with the "naive" parser idea... anyone can write a bad parser, we can't protect against that (and I'd be scared if we could!).

Same goes for the security concerns you bring up. They've been brought up, and even addressed, already. If not in this thread/ticket, than in multiple other ones, which leads me to my last refutation, that being the idea this problem affects a negligible number of people.

It seems to be effecting many people (including authors of popular tools who are representing thousands of people), as you can tell by the various tickets around this issue. The same-ish issue (for MD) has the highest number of comments at the moment, and it's just a bump of several other tickets (all of which seem to have "solved" the problems raised but appear to be in a holding pattern). Plus figure a good 85% of people probably don't file a ticket when they hit a problem...

I'll close by saying that security through obscurity isn't really much protection. There's a reason crypto libs have to be published to be valid, and that's because obscurity is "soft" protection at best, and a huge security concern at worst. It's cool GH is sharing the markup plugin-- tho note numbers 2-5 from the docs ("internal code") seem to be a source of bottlenecks.

Pretty sure The Future will be mostly folks who put source code out so it can be verified/checked (been saying this for years, but at least it doesn't sound crazy anymore... how much does OSS power these days?), and if you don't think someone can just call up their friend who works at Large Company X to get a peek at sources... ¯_(ツ)_/¯ anyhow, at this point it's not a tech barrier as far as I can see, and it's fine if something else is holding this up, but there's been little to no feedback from project maintainers, and it's depressing to see so much input and so many orphaned/ghosted PRs going back years and whatnot (in general around this issue). But not that much. 😃 Mainly I'm just tossing in another +1, in a now age-old tradition. 😜

[mojavelinux]

@obilodeau Thank you for sharing your insight on this issue. You provided a lot of context for what the security concern actually entails.

I want to emphasize two things. First, Asciidoctor has from the very beginning offered up the include processor as an extension point (for GitHub, I might add). That means that GitHub could write code that takes over processing of the include, so there's 0 risk of insecure code that they themselves did not introduce. That doesn't make it easy, to be sure, but that should mitigate the third-party risk argument.

Second, I'm willing to make any reasonable change in Asciidoctor that would further accommodate this feature. So far, I have heard absolutely nothing from GitHub on this issue, which doesn't bode well if we expect to make any progress. My offer stands to work with GitHub if they need me to get this feature implemented. (code, documentation, whatever)

Olivier wrote:
An include can include includes which could include the first top-level include.

I don't expect GitHub to use the built-in include processor, but there is protection against this scenario. An include can only go to a fixed number of levels before it's terminated. Most users would be happy with 3-5 levels, if even that.

Olivier wrote:
AsciiDoc includes can be URIs, URI includes could be used to perform SSRF attacks in order to explore GitHub's infrastructure in means that shouldn't be allowed.

I would not except these to work on GitHub. They don't even work on Asciidoctor without a command-line / API flag. I think includes that work within the set of repository files is already a huge boon for us AsciiDoc writers.

Olivier wrote:
Implementing a hard feature for <1% is not worth the resources (or the risk).

If they went with that argument, then I'd say they don't know the AsciiDoc community very well. AsciiDoc users are bonkers for includes, rightfully so. As Denny points out:

It seems to be effecting many people (including authors of popular tools who are representing thousands of people), as you can tell by the various tickets around this issue.

If you want my prediction on this issue, it will be resolved as soon as GitLab implements it. That's not trolling, that's just sound competition in the marketplace. After all, GitLab just implemented stem support (i.e, math expressions) in AsciiDoc files.

[mojavelinux]

Btw, here's a link to corresponding issue in the GitLab issue tracker.

https://gitlab.com/gitlab-org/gitlab-ce/issues/18045

Interesting to note that @jirutka has already submitted a full patch for a custom include processor that works with a repository manager like GitHub and GitLab. So the code is there. We just need the will.

[jyutzler]

I am fine with a very restrictive policy here, even restricting includes to the same repository. My issue is that I have documents that I want modularized but I also want to be able to present them on the web as complete documents and right now I can't do that. I have to kick off a separate process to generate the file then post that. Whether I do this manually or through some sort of agent is irrelevant, it is a nuisance and completely unnecessary.

[kavaliro]

3-5 levels would be perfect imho. Five gives a few levels for document structure and a couple of levels for templates. Three would suffice for most use cases, but five would handle almost all of them.

While I can handle all of the includes before pushing the content to github, effectively that means I'm no longer able to use github as source control (for the same reason our .gitignore files exclude /bin.)

One use case I have uses include to add an svg, and the text fields of the svg are then changed via substitutions. Which really isn't that complex--an svg is just xml after all--but it's not something you can do with the standard image handler.

Count me among those who are bonkers for includes. Includes really are a must have feature for asciidoc.

[denuno]

Heck, I'd settle for even one level-- and a same-repo policy is what I'd expect off the top of my head so that'd be dandy too (though it'd be swell if they follow the standard GH route of an inter-github policy for linking to stuff in other GH user/org repos).

While I can handle all of the includes before pushing the content to github, effectively that means I'm no longer able to use github as source control (for the same reason our .gitignore files exclude /bin.)

☝️ this!

One of the main reasons I like GH is that slickly rendered README. It is powerful feature. Without includes it's hobbled in exactly the way you wouldn't want though, as outlined above (making it a non-option for folks who have organized docs in their sources ("Don't look at those, look at the generated ones! (but yes, edit those)" yuck).

GH has had includes of various types (at least header/footer) for years now in the wiki section, so it's a problem that was solved for GH in some general form ages ago...

[kavaliro]

I just want to mention that two workarounds are possible for this, for those needing a solution.

1. There's a jekyll asciidoc template floating around that is set up to use Travis-CI to do the compilation, committing the results to the gh-pages branch of a repo.

2. If you need to retain tighter control and compile locally, you can use git hooks to kick off scripts to do that, commit to the gh-pages branch, and push it. (I haven't tested that, but i don't see any reason it wouldn't work at first glance.)

[mojavelinux]

Unfortunately, publishing to GitHub Pages, or Netlify, or various other static web hosts, is a different solution entirely. If you use your own static site generator, you open up a whole world of possibilities. But the README and other files in a repository on GitHub are still crippled.

A better workaround is to have a CI job that monitors for changes to AsciiDoc files in the repository and expands the include directives, obviously leaving behind a hint so that the region can later be updated. It's not ideal, but it at least lets you keep your documents DRY.

[denuno]

Yeah, the closest thing to a work-around would be some type of include that works now, like images, but that's a downer for various reasons.

I can whip up a PR for this project if there's even a whiff of interest from someone in powah, but this is a case where I won't be able to use my own fork, so I'm less than motivated to write it unless there's at least a chance of it getting in... there's been nothing on most these issues, and I'm, let us just say, not optimistic, about [email protected] being able to help.

Maybe @MikeMcQuaid can point us in the right direction?

[jyutzler]

I fully understand that publishing to a static web host is a different matter and I will be sure to take a look at jekyll-asciidoc to see if it is a potential partial solution. It doesn't change the value of rendering AsciiDoc directly through GitHub (as is done with Markdown, GeoJSON, etc.).

[MikeMcQuaid]

I can whip up a PR for this project if there's even a whiff of interest from someone in powah, but this is a case where I won't be able to use my own fork, so I'm less than motivated to write it unless there's at least a chance of it getting in... there's been nothing on most these issues, and I'm, let us just say, not optimistic, about [email protected] being able to help.

Maybe @MikeMcQuaid can point us in the right direction?

@denuno This isn't something I have or do work on at GitHub. Please email [email protected] rather than @mentioning me unless it relates to my open source work. Thanks!

[denuno]

Apologies, I grabbed the first person who looked to have some type of power to do something for the project, maybe I should have @ more prolific committer-- or it seems like maybe I have no idea how contribution works for github projects on github? 🤔

It would seem odd if, it weren't par for the course (I'm the same I reckon), that github projects don't use github, and instead have some out-of-band deal going on... seriously though? Hit up a generic 'support' addy for feedback on issues? 🙄

I sent an email. I shouldn't have bitched before trying it, but regardless-- that there is a disconnect which is worse than missing include functionality! Consider this comment a PR to address it by connecting projects to people (or at least positions beyond 'support'). 😛

[denuno]

I'd be remiss not to do a PR now. 😃

Is anyone else already on it? Is there anything really to do, for that matter? This stuff just calls the external tools anyhow, not sure why any special things needed to be added in the first place?

I thought there was more to it, because of all the talk about security concerns being the main reason includes have not been implemented, but if there are security concerns with this set up, github is exposing that the infrastructure is way more vulnerable than it should be...

Like, does this really get run on some type of master server that has write access to all github repositories or some such? Could I craft an Evil Commit that would Do Something to other repositories? Is servable content + traffic not watched to verify content isn't misbehaving?

There must be a ton of stuff checking for more issues than I can think of off the top of my head... what is the real reason this hasn't been implemented in 5+ years?

If it really is a security concern, please have [email protected] contact me privately, and I'll point out the attack vectors that are exposed by stating such (which are pretty obvious, and have nothing to do with includes, so I'm hoping that idea was just someone's attempt to keep things simple -- too simple, sure -- but a worthy thing to strive for... or maybe it was some misdirection... I could get behind that too 🕶️).

😆 I meant to just say I'd put my money where my mouth is, versus piling on some more, but I started really wondering mid-type.

[denuno]

Just for some closure on this:

I dropped [email protected] a line, and someone got back to me pretty quick!

Includes are on "The List" (for next year), and there's nothing anyone outside GitHub itself can do or contribute to speed that up. It's a coderpower issue. To be on The List even a year out is like "yay!". 😝

For anyone looking for authoritative feedback:

🔥 Contact [email protected] to get information about issues in the GitHub issue system.

We can then person-power the [shadow] issue systems on github by relaying responses, as I'm doing now.

Ha! Just had the awesome idea to write a GitHub support GitHub issues integration. Once you automate a shadow system, it's technically some type of mirror, right? 🤔

[lukehefson]

Hey folks 👋. I'm a PM at GitHub working with some of our Render folk and I've found this thread via
https://twitter.com/matthewmccull/status/1083619220858986497

In truth, we've got some infrastructure work to pay down first before we can take a serious look at this again, but I want to let you know that it does matter to us. We're hoping to make inroads soon into that infra work – and then we can start investigating how best to address this for y'all!

cc/ @mojavelinux @jexp
FYI @matthewmccullough @clarkbw @skalnik

[jexp]

That would be amazing, thanks so much @lukehefson

Btw. I love your "small-ux-wins" (papercuts) project, that has already improved my GH experience a lot.

[Kukulkano]

Huh, I just found this after my AsciiDoc documentation does not include the glossary. Can't believe this is not working.

If I read the comments, they want us to pre-process the documentation just to generate a monolithic file because GitHub does not fix that issue since 5 years now?

I will definitely not run a Makefile or a GitHub action just to get around this problem. I was recommended by colleagues to switch to GitLab some time ago. I might consider that again. So sad...

[gautaz]

For people who still insist on using Asciidoc with github :-), here is an example of using the asciidoc reducer that @mojavelinux advised for: https://github.com/r2d2bzh/docker-build-nodejs/blob/848e41ba6249de933816bce29400606594871e59/.github/workflows/readme.yaml

This is in fact not an ideal solution (one cannot edit the readme file directly and it also increases - slightly - the footprint of the git repo) but at least it seems to work.

[jerstlouis]

This is a major pain and reason to migrate to GitLab for standard editors using asciidoc e.g., with metanorma.

[denuno]

I swear I just saw someone add a comment that I don't see here now, about how this feature is also needed to load attributes— which I couldn't agree more with!

I was able to cut out a ton of boilerplate by using .asciidoctorconfig files in my docs. (I'd make a separate issue for that feature were this to get implemented and not have that capability.)

GitLab has my docs looking beautiful! The .asciidoctorconfig file works y todo! It's super nice to just plop docs in and have them look, at least structurally, similar to the rendered docs (which are so, so lovely! Asciidoctor is hands-down the best documentation format developers have available to them so far!).

Being able to include files, and tagged segments of source code, etc. cuts down on so much duplication.

GitHub, what can we do to help make this happen? How can we add this issue to https://github.blog/changelog/label/issues/? It's neat how JetBrains uses YouTrack… FWIW it would be really cool to see GitHub eating it's own dogfood. It looks like that issues link has content generated from an out-of-band system. Maybe an separate special issues system opens up when you install the enterprise edition?

I'll maybe try hitting up support again, just to see if anything is different there, now that Microsoft has taken over [side note: proud of MSFT for not being the Evil Empire anymore 🥰 really dig the move towards open source and collaboration!]. If things are about the same, maybe I'll go ahead and finish that shadow system to link these user-created issues to [email protected]. ;]

I'm also trying to do the meme— where the guy gets hired, fixes a bug, and then quits — which I will 100% do, so know that that job application, where I talk about this issue, is for reals legit.

I'm mostly just saying all this to the audience here (I don't think GitHub folks really even check these), so keep the faith folks… someday this core documentation feature will be added— it must! It's the only acceptable end for this issue!

As always, best wishes everyone! I never mean to be more than silly— but text isn't my first language. Heh.

[vintprox]

I swear I just saw someone add a comment that I don't see here now, about how this feature is also needed to load attributes— which I couldn't agree more with!

Ah, that was me. I had some doubts about the value of my comment and removed it. Turns out it somehow had some gears moving. 😄

I'm working on a documentation, where I wanted to add few repeated sections, sort of a template with shields.io pretty badges and swag. Unfortunately, it's a lot of repetition to just casually copy-paste and expect contributors be on the same wavelength to not wreck it.

Have to either maintain large monolith file or make "sense" of include-as-links (the less the better) - all to please GitHub's limited flavor of Asciidoctor converter.

I wouldn't even think we have a great option to move to. Until Gitea instances come with Asciidoctor converter activated out-of-the-box, without lifting from admins, it's all roses on GitLab side. But GitLab is pretty controversial to me, and I don't think I'll enjoy this migration for asciidoc thing alone.

[toastal]

Codeberg copied Microsoft GitHub's lack of imports and doesn't render any better. I wish they took GitLab's lead as you can link in files like Mermaid docs to keep them separate. While AsciiDoc isn't supported by default on SourceHut, at least you can publish any arbitrary HTML to the README.

[denuno]

I wouldn't even think we have a great option to move to. Until Gitea instances come with Asciidoctor converter activated out-of-the-box, without lifting from admins, it's all roses on GitLab side. But GitLab is pretty controversial to me, and I don't think I'll enjoy this migration for asciidoc thing alone.

I hear ya on GitLab, it's kind of a shame they've gone the route they have, "open-core" where the core lacks some common features to get you to pay, versus getting you to pay to get some awesome extra features (which they have plenty of). Not sure if that's why it's controversial to you, but that's why GitLab is controversial to me.

That said, out of the box it has all anyone needs, really. For sure a drop in replacement for GitHub. I kinda like their CI stuff more than GitHub Actions, after working with both for a while.

Gitea with Drone CI is a pretty complete solution too, and there are some docker images that have everything setup up, including the Asciidoctor renderer. But you do need to have admin powers or an admin who's willing to fire up containers for you.

It's just too bad GitHub doesn't really support asciidoctor, after the author even made changes to help make it happen. Folks like to use this place as their portfolio, so it would behoove GH to make that experience the best possible, to support heads, versus having an objectively bad UX as someone's first experience with a project's README.

Perhaps none of the GH peeps have tried writing docs using Asciidoctor, and that's why they don't have much motivation. Or maybe they don't use GH, and see it as a product they sell, versus a tool they use and happen to sell as well. They're probably required to use an internal Azure DevOps setup (not bad either, but at the other end of the spectrum (whereas GH promotes FOSS front and center)). Too bad that one dude who was inside was never able to make this happen. I've been there myself, so no bad vibes, just like, "alas" ones. 😀

[QumeaJR]

Hey folks 👋. I'm a PM at GitHub working with some of our Render folk and I've found this thread via https://twitter.com/matthewmccull/status/1083619220858986497

In truth, we've got some infrastructure work to pay down first before we can take a serious look at this again, but I want to let you know that it does matter to us. We're hoping to make inroads soon into that infra work – and then we can start investigating how best to address this for y'all!

cc/ @mojavelinux @jexp FYI @matthewmccullough @clarkbw @skalnik

any update on that "does matter" statement from >2.5years ago @lukehefson ?

[jseguillon]

Hey nice to hear you're taking care about this issue ! Please let us know as soon as you got a plan to solve it 🙏

[LUC18fknU7P]

Hey folks 👋. I'm a PM at GitHub working with some of our Render folk and I've found this thread via https://twitter.com/matthewmccull/status/1083619220858986497
In truth, we've got some infrastructure work to pay down first before we can take a serious look at this again, but I want to let you know that it does matter to us. We're hoping to make inroads soon into that infra work – and then we can start investigating how best to address this for y'all!
cc/ @mojavelinux @jexp FYI @matthewmccullough @clarkbw @skalnik

any update on that "does matter" statement from >2.5years ago @lukehefson ?

He has quit Github as an employee a long time ago. Lol. MSFT and Github are lazy AF so don't expect anything from them. In these big companies approx 0.01% of engineers are ACTUALLY working.

[ryandeussing]

@denuno

GitLab has my docs looking beautiful! The .asciidoctorconfig file works y todo!

It does? Can you share a repo? I have my .asciidoctorconfig working with my editor extension (asciidoctor-vscode, pre-release), but in GitLab none of my attributes are defined unless I define them inline, or import a file that defines them, which is what the .asciidoctorconfig is supposed to obviate. Thanks.

[snowe2010]

I think it is probably pointless to make more comments on this thread. It seems to me like they've completely abandoned this repository and so all this shouting is just going into the void. Best to go comment on social media, or hound their support email. I seriously doubt anyone from the company is reading a single one of these issues.

[rawkode]

I wonder if this would be possible now with GitHub Blocks?

https://github.com/githubnext/blocks

Edit: Never mind, they killed it.

https://gist.github.com/idan/325676d192b32f169b032fde2d866c2c#github-next--technical-preview-sunsets

[SaschaWillems]

This gets a +1 from me. We're using Asciidoc + Antora for a new documentation project for the Khronos group, and use includes to avoid duplicate documentation. But those includes are simply ignored when viewing files through github.