How to build the __site locally and deploy on GitHub as a project?

[yenson-lau]

I think this is a very desirable feature. For example, if I generate some complex plots for my websites using a variety of Julia packages, I don't want GitHub to have to reinstall all the packages from scratch and rebuild the code every time. Instead I already have a completely built __site directory that should be directly deployable.

The problem with the project page, of course, comes down to the prepath. Although I've set the prepath, removed __site from my .gitignore, and ran the following code on my own computer

using Pkg; Pkg.activate("."); Pkg.instantiate();
using NodeJS; run(`$(npm_cmd()) install highlight.js`);
using Franklin;
optimize()

and I've replaced my GitHub action as follows

jobs:
  deploy-site:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/checkout@master
      with:
        persist-credentials: false

    - name: Deploy
      uses: JamesIves/github-pages-deploy-action@releases/v3
      with:
        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        BRANCH: gh-pages
        FOLDER: __site

I still couldn't get the site to deploy correctly. Is there any way for me to build the correct site on my own computer? Thanks!

[yenson-lau]

I realized the answer was in the Deploy.yaml file the whole time! Just check keywords for optimize().
This build script will check the config for the pathspec and send it to optimize(). Run, then push.

using Pkg; Pkg.activate("."); Pkg.instantiate();
using NodeJS; run(`$(npm_cmd()) install highlight.js`);
using Franklin;
import Franklin;

rm("__site", recursive=true, force=true);

Franklin.def_GLOBAL_VARS!();
Franklin.convert_md(read("config.md", String); isconfig=true);
optimize(prepath=Franklin.globvar(:prepath));

[yenson-lau]

Unfortunately the highlight.js issue makes running install hightlight.js above pretty pointless. To overcome it I had to delete all the compiled / package files and reinstall Franklin if FD_CAN_HIGHLIGHT==false (not sure if it's necessary to delete the package files).

using Pkg; Pkg.activate("."); Pkg.instantiate();
import Franklin;

if !Franklin.FD_CAN_HIGHLIGHT
  # This whole bit exists simply because the constants in Franklin
  # can't be updated even after highlight.js is installed

  # e.g. /home/user/.julia
  juliapath = pathof(Franklin);
  juliapath = juliapath[1:findfirst("packages", juliapath)[1]-2];

  Pkg.rm("Franklin");
  rm("$(juliapath)/packages/Franklin", recursive=true, force=true);
  rm("$(juliapath)/compiled/Franklin", recursive=true, force=true);

  using NodeJS; run(`$(npm_cmd()) install highlight.js`)
  Pkg.add("Franklin")
  run(`julia build.jl`)

else
  # Once highlight.js is installed
  rm("__site", recursive=true, force=true);

  Franklin.def_GLOBAL_VARS!();
  Franklin.convert_md(read("config.md", String); isconfig=true);
  Franklin.optimize(prepath=Franklin.globvar(:prepath));
end

This is a pretty messy workaround that involves running a new instance of Julia from the shell. Considering that the issue's been raised since August it really needs to be fixed.

Even if most people only intend to use Franklin for simple blogs and wouldn't mind rebuilding the site every time via a GitHub action, the whole point of Franklin is that it can embed Julia code, code that can become too cumbersome to be rebuilt on GitHub every time.

[tlienart]

Hello @yenson-lau I'm reopening as I just want to confirm a few things if you have the time:

• did you manage to deploy from your local __site folder setting the prepath etc?
• did you consider pushing your __site folder (so removing that from the .gitignore) and pointing GitHub pages to it? (i.e. indicate that the folder you want to deploy is __site), if you do that you can discard the deploy.yml, just do stuff locally, push it and let GitHub look at the right path, the only thing you'll need to do is to specify the prepath as you already know but this won't require re-running all the Julia code)

I'm not 100% sure I follow the issue with highlight.js, is it an installation problem where Franklin tells you that it doesn't find the package?
Note that you can always ignore this whole pre-rendering step and either serve highlight.js yourself or use a CDN. While I personally prefer pre-rendering because it decreases the amount of javascript on any one page + some people disable it, this is mostly a matter of taste.

Thanks for the detailed reports in any case, it's very useful to get those even if I can be slow to fix things

[yenson-lau]

Hi @tlienart sure thing, happy to provide as much info as I can and thanks for checking in to this!

• When I'm developing I just use the serve command so the site is built without using the prepath, but before I push to GitHub I run optimize() using the prepath and that works
• Yep I disabled __site from .gitignore. But unfortunately GitHub pages only allows you to deploy from the root directory or /docs, so you still need to move everything from __site to either of these before or after pushing. I kept Deploy.yaml so I can move the site to root on the gh-pages branch after.

The highlight.js problem basically proceeds as follows

1. First add Franklin to a new Julia installation (just so I can assume the dependencies are fresh for Step 2)
2. Then when you try to optimize(), Franklin will complain that highlight.js isn't found. The problem is that the NodeJS.jl uses its own npm, and that one hasn't installed highlight.js
3. So let's say we run(`$(npm_cmd) install highlight.js`), then open a new instance of Julia
   • run(`$(nodejs_cmd() -e "require('highlight.js')"`) will then be successful
   • however, if you run optimize(), Franklin will still complain that highlight.js hasn't been installed

The reason for this is simply that the const FD_CAN_HIGHLIGHT from build.jl remains false after installation, because the package has already been compiled.

The quick and dirty fix is probably to try run(`$(npm_cmd) install highlight.js`) if shell_try(`$NODE -e "require('$(HIGHLIGHTJS[])')"`)==false during the build, before setting const FD_CAN_HIGHLIGHT. This should take care of most instances of this problem.

[tlienart]

ok great I think both points can have quick fixes:

1. have an example deploy script which takes the content of __site without changing it and puts its content "as is" on gh_pages
2. make FD_** part of FD_ENV which is runtime.

I'll check both these things as soon as I have some time :)

[hustf]

I also find that serve() works locally, but online rendering with optimize doesn't work in the same way with my local katex functions.

Making an alternative installation and operative system work with optimize() and online isn't as inspiring as it is time-consuming. I look forward to example 1, deploying the local files directly.

[tlienart]

@hustf in the meantime could you try disabling prerendering? So change the call to

optimize(prerender=false)

[hustf]

Thank you for the tip. I retried that, but though I struggle with prerendering and NodeJS, that's less important in my case.

I have a quite fragile setup for rendering the equations which is likely to be the problem. 'utils.jl' loads Latexify.jl etc., and defines some ad-hoc code. It works locally with using Franklin;serve(). This is on windows, with no administrator privileges. For some reason which I can't figure out, it works slightly different online. I, too, noticed that, online, Franklin.FD_CAN_HIGHLIGHT = false. And so, in this version, NodeJS is dropped and prerender=false.

The gh-pages files contains, typically:
\[\frac{du}{dt} = \alpha \cdot u\left( t \right) = 1.01 \mathrm{ \textcolor{#15a}{s^-1}} \cdot u\left( t \right)\]

Correct version, as rendered locally by functions defined in utils.jl:
\[\frac{du}{dt} = \alpha \cdot u\left( t \right) = 1.01 \mathrm{ \textcolor{#15a}{s{^-}{^1}}} \cdot u\left( t \right)\]

Screenshot from online version:

Screenshot from local version:

My current deploy.yml:

name: Build and Deploy
on:
  push:
    branches:
      - main
jobs:
  build-and-deploy:
    runs-on: ubuntu-latest
    steps:
    - name: Checkout
      uses: actions/[email protected]
    # NOTE: Python is necessary - for the pre-rendering (minification) step

    - name: Install python
      uses: actions/setup-python@v2
      with:
        python-version: '3.8'
    # NOTE: Here you can install dependencies such as matplotlib if you use
    # packages such as PyPlot.
    # - run: pip install matplotlib

    - name: Install Julia
      uses: julia-actions/setup-julia@v1
      with:
        version: 1.6
    # NOTE
    #   The steps below ensure that NodeJS and Franklin are loaded then it
    #   installs highlight.js which is needed for the prerendering step
    #   (code highlighting + katex prerendering).
    #   Then the environment is activated and instantiated to install all
    #   Julia packages which may be required to successfully build your site.
    #   The last line should be `optimize()` though you may want to give it
    #   specific arguments, see the documentation or ?optimize in the REPL.
    - run: julia -e '
            using Pkg; Pkg.activate(".");Pkg.Registry.add("General");
            Pkg.Registry.add(RegistrySpec(uuid = "23338594-aafe-5451-b93e-139f81909106"));
            Pkg.Registry.add(RegistrySpec(url = "https://github.com/JuliaRegistries/General.git"));
            Pkg.Registry.add(RegistrySpec(url = "https://github.com/hustf/M8.git"));
            Pkg.add(["Franklin", "Latexify", "MechanicalUnits"]);
            Pkg.instantiate();
            using Franklin;
            optimize(;prerender=false)'
    - name: Build and Deploy
      uses: JamesIves/[email protected]
      with:
        branch: gh-pages
        FOLDER: __site

[tlienart]

could you try

optimize(; prerender=false, minify=false)

and report? (this will, in effect, be identical to your local setup unless you somehow have pinned versions of packages) failing that, would you mind adding me to the repo with your website so I can have a look?

[hustf]

Done. The output can be confusing due to ad-hoc debug code from my parts, but the julia run ends like this:

→ Use Pkg.activate() to go back to your main environment.                       

→ Full pass (no pre-rendering)     [done  25.4s]     

I'll add you to the repository at once, and here's the last run. https://github.com/hustf/DEQ/runs/2793296994?check_suite_focus=true

Also logging on to slack/Franklin.

[tlienart]

ok I think this is quite a different issue. I don't think it's related to OP (I'm not even sure it's Franklin). My hypothesis is just that there's a problem with ⁻¹ which gets considered as one or two characters possibly based on the platform (this is just a guess based on the result where you either get ^-1 or {^-}{^1} not sure at which point this happens, possibly in your modify_latex function).

The following is maybe not 100% satisfactory but should fix they issue and be more robust:

\newcommand{\inv}{^{-1}}

and then 1.010s\inv etc. (or you could just type ^{-1} in full.

(I'd suggest opening another issue if you'd like to discuss this further as I don't think this is related to the path discussed in the rest of the thread).

Ps: good stuff with all these hfuns, I'd be happy to have feedback in a separate issue if you wished there were additional things to help people develop and test those (e.g. to avoid what you stumbled upon)