Setting up highlight.js for deployment manually

[mbaz]

I am using PkgPage to write the documentation for a small package. The docs will be hosted in Github, under mbaz.github.io/GtkServer.jl/. I am using serve(single=true) to generate the website, and then I push the contents of page/__site to a gh-pages branch on Github.

However, after running serve(single=true), the paths in page/__site/index.html don't include the prepath defined in config.md.

This line is in my config.md:

@def prepath     = "GtkServer.jl"

All the paths in index.html are missing the prepath; for example:

<link rel="stylesheet" href="/libs/highlight/styles/atom-one-dark.css">

As a result, the webpage renders incorrectly. This happens for all paths: javascript, css, libraries and assets are all affected.

Note this line in config.md produces identical paths:

@def prepath     = ""

If serve() should not add the prepath, is there some other Franklin or PkgPage function to add it?

[tlienart]

Hello!

The easier route would be that you use GitHub actions but I understand if you’d rather not.

If you have a look at the deploy script, you’ll see that the last command before deploy is optimize; if you call that then the stuff in __site will have its paths fixed & you can push it as you were doing

Here’s the line:

PkgPage.optimize(input="page", output="")'

Where the args indicate where the folder is (if you use the default leave as is) and should be (if you just want it to be in gh-pages as is, leave blank)

[mbaz]

Thanks! However, I got a warning regarding highlight.js:

julia> PkgPage.optimize(input="page",output="")
┌ Warning: I couldn't load 'highlight.js' so will not be able to pre-render code blocks. You can install it with `npm install highlight.js`.
└ @ Franklin ~/.julia/packages/Franklin/msAXw/src/manager/post_processing.jl:169

I installed highlight.js like this, running from my project directory (not inside page):

julia> run(`sudo $(npm_cmd()) install highlight.js`)
npm WARN saveError ENOENT: no such file or directory, open '/home/miguel/rcs/GtkServer/package.json'
npm WARN enoent ENOENT: no such file or directory, open '/home/miguel/rcs/GtkServer/package.json'
npm WARN GtkServer No description
npm WARN GtkServer No repository field.
npm WARN GtkServer No README data
npm WARN GtkServer No license field.

+ [email protected]
updated 1 package and audited 42 packages in 0.737s
found 0 vulnerabilities

Process(`sudo /home/miguel/.julia/artifacts/a42f5c72a500a8683c08a5e818d27a104fc9cd5a/bin/node /home/miguel/.julia/artifacts/a42f5c72a500a8683c08a5e818d27a104fc9cd5a/bin/npm install highlight.js`, ProcessExited(0))

Do you know what the problem is?

---

Update: running from my project directory (not under page):

    $ node -e  "require('highlight.js')"
    $ echo $?
    0

I'm trying to figure out why Franklin sets FD_CAN_HIGHLIGHT to false.

---

$ julia
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.4.2 (2020-05-23)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> import NodeJS

julia> const NODE = begin
           if "NODE" ∈ keys(ENV)
               ENV["NODE"]
           else
               NodeJS.nodejs_cmd()
           end
       end
`/home/miguel/.julia/artifacts/a42f5c72a500a8683c08a5e818d27a104fc9cd5a/bin/node`

julia> shell_try(com)::Bool = try success(com); catch; false; end
shell_try (generic function with 1 method)

julia> const FD_CAN_HIGHLIGHT = shell_try(`$NODE -e "require('highlight.js')"`)
true

julia>

However, it looks like Franklin sets this variable to false for some reason.

---

If I cd to Franklin's directory (~/.julia/packages/.../src) and I run

julia> using NodeJS
julia> include("build.jl")
true

From my project's directory:

julia> const FD_CAN_HIGHLIGHT = shell_try(`$NODE -e "require('highlight.js')"`)
true

I'm completely stumped about why Franklin can't find highlight.js. Please let me know if you need me to run any tests.

[tlienart]

This is the kind of reason why using GitHub action is easier :-)

You need to install highlight.js in a way that Julia can find it (good to do it via NodeJS) and then need to rebuild Franklin.

Note that if Franklin complains about not finding these libs, it’s totally fine, it just means that your JavaScript for code and maths will not be pre-rendered, this is not an issue and your page will still render fine.

[mbaz]

Thanks Thibaut, but I'm still confused.

• I'm installing highlight.js the way it's recommended in Franklin's and PkgPage's documentation, which means that Julia should be able to find it, right?

• If I run Franklin's build.js by itself, it does find highlight.js, but it doesn't find it when I run optimize. Is that because higthlight.js is not installed correctly?

If you think this is not a bug, please let me know and I'll take the discussion to Julia's forum -- I don't want to pollute your issue tracker with support questions.

[tlienart]

Hmm sorry I think it’s important to get what’s going on here and improve the docs.

Can you just clarify what you did when you say you ran the “build.js”? Can I just check with you that you did

1. The NodeJS install as per the doc
2. A rebuild of the Franklin package with Pkg.build(“Franklin”)

and that after restarting a Julia session and doing using Franklin you still had the warning?

Thanks a lot!

[mbaz]

I finally hit on the right incantations and it seems like everything is working. Thanks for your help!

One final comment: the documentation for PkgPage suggests running

run(`$(npm_cmd()) install highlight.js`);

whereas Franklin's suggests running

run(`sudo $(npm_cmd()) install highlight.js`)

(note the sudo). Which is the preferred way, and could the documentation be more consistent?

[tlienart]

Great, I’ll leave this open as a reminder to fix the docs and make it clearer what the steps need to be, thanks

[mbaz]

@tlienart I think one reason why this ended up being so confusing for me is that running ]build Franklin will re-run the checks for the extras, but it doesn't force a precompilation, and thus it doesn't update the actual Franklin code run by Julia. That is why Franklin kept reporting FP_CAN_HIGHLIGHT = false, but when I ran the same code on the REPL, I got true.

After I triggered a recompilation (by ]dev Franklin and modifying a file), then things started to work.

Do you want me to open an issue over in Franklin's tracker?

[tlienart]

It’s fine here or there I’ll adjust the docs as per your comments, thanks for the feedback, much appreciated!

Ps: as a side note I also have experienced such issues with NPM, it’s sometimes a bit weird where assets are installed and how to indicate that to Julia, the word “incantation” works well; maybe we could provide an automated process for this but it would require a better understanding of NPM

[mbaz]

I'll try to write down the steps to get the extras (highlight.js and purgecss) installed and operational.

Also, I think it would be worthwhile to include a mechanism in Franklin to detect the extras if they are installed after Franklin. Right now, the detection code only runs when Franklin is precompiled, and there is no easy way to trigger a precompile. One way to solve this would be to detect the extras at runtime; another would be not to declare FD_CAN_HIGHLIGHT and similar as constants. Possibly a more difficult option would be to modify build.jl to trigger a precompile (so the user can just run ]build Franklin).

[mbaz]

Quick update: so far I've been unable to figure out a sure-fire way to install the extras in a repeatable manner in a global directory. npm is infuriating: it's the only installer I've ever used that can install a package and then be unable to find it. I'll keep working on it.

---

@tlienart Do you want to allow users to have a global npm installation directory? Personally, I think it's much better than having to install the modules in every page directory. However, right now PkgPage looks for purgecss inside page/node_modules. Adding -g to npm root allows for a global installation (line 109 in instantiate.jl):

run(pipeline(`$(NodeJS.npm_cmd()) root -g`, stdout=io))

---

Update: purgess does not run if node is not installed (its first line is #!/usr/bin/env node). So, some parts of PkgPage use Julia's node artifact while others use the system's node.

Would it be acceptable to set things up to use the system's node binary if it is present?

---

Update 2: I got it to work. Specifically, I know how to make it work with a global installation of nodejs modules, and with the system's (not Julia's) nodejs install. I need to take a break, I'll come back later with more details.

[tlienart]

Thanks so much for this effort, hugely appreciated

[mbaz]

Thibaut, here are my notes. These are broad and cover more than this single issue. Let me know if these need clarification, or if you want me to submit separate issues or anything. I don't know how and/or if you'll decide to tackle these issues, but please also let me how can I help. Thanks!

---

Notes on manual deployment with PkgPage and Franklin, and in particular the installation of highlight.js and purgecss (the "extras").

Current issues or problems:

• PkgPage and Franklin mostly rely on nodejs and npm artifacts. However, purgecss relies on a locally-installed nodejs. This results in an inefficient duplication of binaries.

  Evidence: the first line of purgecss is #!/usr/bin/env node. Since the node artifact is not in the system's PATH, purgecss fails to run without a local nodejs installation.

  Possible solutions: (A) require a local nodejs installation and don't use the artifacts at all. (B) ship a modified purgecss that runs the artifact binaries.

• PkgPage/Franklin fails to detect and use any needed program or module if these are installed after the packages are used in Julia for the first time. The reason is that constants (such as FD_CAN_HIGHLIGHT) are hard-coded when the package is first precompiled, and are not updated again. Running Franklin's build.js script will detect any changes in installed extras, but fails to update the precompiled image.

  As a result, the only way to get Franklin to recognize a newly-installed extra is to uninstall it, remove it from the filesystem, and install it again.

  Possible solutions: (A) don't declare these as constants; (B) provide a mechanism to force package precompilation.

How to install extras:

There are two ways to install the extras:

• Under the page directory (in parallel with __site, etc). This works out of the box, but it has several drawbacks, not the least being that the extras need to be installed N times for N packages.

• Globally (meaning per-user, not per-system). This allows one single installation to be reused for every PkgPage/Franklin project. These are the required steps (assuming Linux):

  1. Modify the PATH environment variable:

  export PATH+=:~/.npm-global/bin
export NODE_PATH=~/.npm-global/lib/node_modules

  2. Install each extra with the -g switch:

  npm install -g highlight.js
npm install -g purgecss

  3. Verify that these are found by node (the commands should produce no output):

  node -e "require('highlight.js')"
node -e "require('purgecss')"

Bundle everything?

Given how small some of these packages are, I think that bundling everything with PkgPage/Franklin is an option worth considering. The included files could even be adapted to this use case; one example would be modifying purgecss to use Julia's node instead of the local install.

[tlienart]

Thanks a lot for this; I'll need to look at this in details and see what's the best way forward (if anything, better docs).

Note also that things "just work" if you use GitHub or GitLab CI which is what most users do, so I'll need to make sure whatever comes out of this doesn't break that.

Thanks again for looking into this!

(Ps: I'm not very keen on the bundle everything path)

[mbaz]

You're welcome! I understand that your priority is using automated CI to deploy. What I'm hoping for is that manual deployment remains an option, for those of us who prefer to work that way.