External renderers and included files

[gautaz]

• Gitea version (or commit ref): Gitea version 8670dec built with: bindata, sqlite
• Git version: git version 2.15.3
• Operating system: Alpine 3.7
• Database (use [x]):
  • PostgreSQL
  • MySQL
  • MSSQL
  • SQLite
• Can you reproduce the bug at https://try.gitea.io:
  • Yes (provide example URL)
  • No
  • Not relevant
  • Did not try yet
• Log gist:

Description

Hello,

I'm currently using asciidoctor to render asciidoc readme files. These files contains include directives that the current process is unable to render correctly. The failure message displayed is Unresolved directive in <stdin> - include::filepath[]

My current understanding is that asciidoctor is unable to find the included file from the environment it is run within. This seems pretty obvious as the content to render is given through the standard input.

Is there a way to circumvent this issue ?

What is the environment in which an external renderer is run within in order to produce the displayed markup ?

[lunny]

Could you have a similiar config with below?

[markup.asciidoc]
ENABLED = true
FILE_EXTENSIONS = .adoc,.asciidoc
RENDER_COMMAND = "asciidoctor --out-file=- -"
; Input is not a standard input but a file
IS_INPUT_FILE = false

[gautaz]

Could you have a similiar config with below?

[markup.asciidoc]
ENABLED = true
FILE_EXTENSIONS = .adoc,.asciidoc
RENDER_COMMAND = "asciidoctor --out-file=- -"
; Input is not a standard input but a file
IS_INPUT_FILE = false

This is in fact my current configuration (taken as is from here).

[gautaz]

@lunny Is there a way to pass asciidoctor a path to the project sources by using the --base-dir option?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[gautaz]

Sorry to insist but the current way to integrate external renderers by using the standard input to feed them will not work for any renderer using other files from the project (for instance in case of document sources including project files).

It seems to me that Asciidoctor is just an example among others.

As Gitea is a collaborative project, I also totally understand that this issue cannot be resolved for now unless someone skilled enough has the time to handle it but I do not understand how closing automatically the issue will help.

Am I the only one thinking that the current external renderer integration method cannot cover all use cases?
Perhaps an agreement on this point is necessary as a preamble to adding an item in the Gitea roadmap?

[zeripath]

Why don't you write a script that will set up the environment correctly for ascidoctor?

You can send the stdin to a temporary file run ascidoctor on that or deal with the input file and then cat the output back out.

[gautaz]

@zeripath Are you saying that this script would be executed instead of asciidoctor and will hand the missing elements over to asciidoctor once it gathered all the missing pieces?
How is this script supposed to access the other files of the git repository where the document source is located? This script will get the same data than asciidoctor, that is a standard input with the content of the document source to process but no information about where to get the other parts of the git repository.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs during the next 2 weeks. Thank you for your contributions.

[Jim-Bar]

Interested as well in a solution to this problem. Including other files doesn't work with e.g. AsciiDoc which is unfortunate because it's a great feature. As such, our documentation cannot be properly viewed from Gitea.

@gautaz Thanks for opening this issue ;)

[shomodj]

Also interested in solution for this...

Can we fund this?

[raylee]

I have the same issue, but for a different file format. If the external renderer could be passed the path to the git repository as a substitution or environment variable in RENDER_COMMAND, the rest could be handled outside of Gitea.

Something like

RENDER_COMMAND = "myFormatRenderer -g $GITEA_REPO_ROOT ... -o -"

would work fine. (Or if the user is running on Windows, they could use the %VAR% form for the command.)

The external renderer would then be able to check out the necessary supporting files which are needed.

[lunny]

OK. So I think the requirement is Expose related renderer context information as variables for the RENDER_COMMAND.

[lunny]

Let's make a sample. Assume we have a git directory in a repository like this.

- index.html
- images/logo.png

Then I want to display the logo image in the HTML file if I enabled the third-party renderer for .html extensions.

Is that you want?

[raylee]

For my use case, the renderer needs to get at the git history to show summaries of what changed. So while this bug itself is about related files, I need to rummage around the git history. Hence my request for an environment variable (or command substitution) of a path to the repo.

This would address both my use case (looking back in time), as well as this bug's request (finding supporting files).

[johnnybubonic]

Just hit this one myself. :/ Bummer.

How is the render executed? Is a checkout performed or does it pull the browser-requested file specifically from git? If the former, this should be fairly easy - just provide a RENDER_CMD directive in markup.* of app.ini that provides the directory of the cloned repository as an env var. A couple wrapper scripts or programs and rendering can be performed however desired. I suspect this is not how it's done, however, as that'd be a nightmare to implement for multiple requests and refs in parallel.

If the latter, however... it's not really possible, as gitea itself would need to be parser-aware of each markup language they wanted to support, which defeats the entire purpose of a "modular" system like it currently is designed to be.

So I'd personally probably classify this as WONTFIX because I suspect it's impossible.

For those needing a solution, however, how I've implemented this in the past is simply with a pre-commit hook that just renders to e.g. an HTML file (I'm a heavy user of Asciidoctor). Unfortunate that it then needs to be on every machine though. There are some proposed solutions (including setting DISABLE_GIT_HOOKS to false).
I find that https://github.com/gabyx/Githooks is probably the most elegant approach.

[lunny]

Since you could do that in markdown with ![Alt text](relative_path/img.jpg) to render an image, I don't think this is not impossible for images. We just need to parse image elements and replace the relative link to Gitea's file raw link.
For CSS, Javascript, we need to assess whether there are some security problem.

[johnnybubonic]

Markdown is not the only text markup out there. This bug was opened initially for Asciidoctor.

[lunny]

We already have two environment could be used.

GITEA_PREFIX_SRC, which contains the current URL prefix in the src path tree. To be used as prefix for links.
GITEA_PREFIX_RAW, which contains the current URL prefix in the raw path tree. To be used as prefix for image paths.

see https://docs.gitea.io/en-us/config-cheat-sheet/#markup-markup

[luolong]

The root issue seems to be that when rendering external documents from standard input, the renderer has no idea how to find included resources.

For example, Asciidoc includes allow rendering the content of the included document inline within the rendered document.

When an asciidoctor command line utility starts rendering from stdin and encounters an include: directive, it tries to read the referred document and replace the directive with a rendered content of the included file.

Because stdin has no concept of file paths, the renderer fails (semi-gracefully) and renders the failure to include a requested file.

It seems to me that rendering of documents needs to be extended to allow for external renderer to request documents (based on relative paths) as they process the rendering request.

An example.

Given two asciidoc files:

README.adoc:

= README

include:doc/install.adoc[]

== Rendering

Use `asciidoctor` to render this document.

doc/install.adoc:

== Installation instructions

Install asciidoctor

Then final rendering of the READMe.adoc file should look as as though the stdin contained the following:

= README

== Installation instructions

Install asciidoctor

== Rendering

Use `asciidoctor` to render this document.

[sehari24jam]

Recently I have successfuly configured app.ini to display included asciidoctor files, with some notes.

Let said the ROOT_URL of gitea is https://gitea.my.local.

Then app.ini shall have:

[markup.asciidoc]
ENABLED = true
NEED_POSTPROCESS = true
FILE_EXTENSIONS = .ad,.adoc,.asciidoc
RENDER_COMMAND = "asciidoctor -s  -a docdir=https://gitea.my.local$GITEA_PREFIX_RAW -r asciidoctor-diagram -a data-uri -a allow-uri-read -a showtitle --out-file=- -"
IS_INPUT_FILE = false

[markup.sanitizer.asciidoc]
ALLOW_DATA_URI_IMAGES = true

So the trick is to override docdir attribute with URL pointing to ROOT_URL + $GITEA_PREFIX_RAW variable. Explicit docdir is needed, because running asciidoctor-renderer against stdin would have no idea where the docdir.

And modify the related asciidoc files to have {docdir} as prefix.

Let said README.adoc:

= README

This is main file

include:{docdir}/include/sub.adoc[]

And include/sub.adoc:

== Sub Level
In sub

Adding prefix {docdir} will also work under asciidoctor cli. So doing this will make the files both work in gitea-renderer and asciidoctor-cli.

Repository visibility setting need to be set to non-private (under repo setting).
Due asciidoctor-renderer need to access ROOT_URL$GITEA_PREFIX_RAW/include/sub.adoc without restriction.

Use of gitea user-token (https://token@domain) does not work for asciidoctor-renderer.

[luolong]

That is a clever hack, but adding {docdir} to every link seems onwieldy.

I've been playing around in my head with the idea of implementing a virtual filesystem inside gitea, serving content of the current repository commit directly from the git tree and using that for the asciidoctor, but I am no Go expert and I wouldn't know if that scheme would work with asciidoctor invocation at all.

But if that would work, it would help tremendously with asciidoctor rendering.