Jekyll build/deploy workflow #86

0xdevalias · 2020-07-26T02:35:06Z

Sets up a GitHub actions/workflow to build and deploy the Jekyll site to Github Pages.

fixes Dockerize Jekyll + automate site builds #27

0xdevalias · 2020-07-26T02:50:21Z

I'm not actually sure what the best method for testing GitHub actions/workflows is without just deploying to prod.. I found one project claiming to be able to:

https://github.com/nektos/act

But in attempting to run it in dryrun mode, it looked like it would fail:

⇒  act -n
*DRYRUN* [Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
*DRYRUN* [Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1
*DRYRUN* [Build and deploy Jekyll site/jekyll] Unable to resolve v1: reference not found
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ❌  Failure - 💎 setup ruby
Error: reference not found

I expect if we used an exact version tag it would probably work.. but that seems at odds with how other examples seem to imply GitHub works?

Changing ruby/setup-ruby@v1 to ruby/[email protected] gave the following output, implying it should work correctly:

⇒  act -n
*DRYRUN* [Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
*DRYRUN* [Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1.40.0
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 💎 setup ruby
*DRYRUN* [Build and deploy Jekyll site/jekyll] ⭐  Run 🔨 install dependencies & build site
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/limjh16/jekyll-action-ts' # ref=v2
*DRYRUN* [Build and deploy Jekyll site/jekyll]   ✅  Success - 🔨 install dependencies & build site

Yet when we run it properly, we hit an issue based on the docker image that is being used not matching perfectly to how GitHub actions runners are:

⇒  act
[Build and deploy Jekyll site/jekyll] 🚀  Start image=node:12.6-buster-slim
[Build and deploy Jekyll site/jekyll]   🐳  docker run image=node:12.6-buster-slim entrypoint=["/usr/bin/tail" "-f" "/dev/null"] cmd=[]
[Build and deploy Jekyll site/jekyll]   🐳  docker cp src=/Users/devalias/dev/_sites/devalias.net/. dst=/github/workspace
[Build and deploy Jekyll site/jekyll] ⭐  Run 📂 checkout
[Build and deploy Jekyll site/jekyll]   ✅  Success - 📂 checkout
[Build and deploy Jekyll site/jekyll] ⭐  Run 💎 setup ruby
[Build and deploy Jekyll site/jekyll]   ☁  git clone 'https://github.com/ruby/setup-ruby' # ref=v1.40.0
[Build and deploy Jekyll site/jekyll]   🐳  docker cp src=/Users/devalias/.cache/act/[email protected] dst=/actions/
[Build and deploy Jekyll site/jekyll]   ❗  ::error::ENOENT: no such file or directory, open '/etc/lsb-release'
[Build and deploy Jekyll site/jekyll]   ❌  Failure - 💎 setup ruby
Error: exit with `FAILURE`: 1

So it seems we may not actually have a good way to be able to test this locally..

0xdevalias · 2020-07-26T03:05:27Z

The build failed since the runner doesn't have the appropriate gsl libraries installed (required to speed up --lsi)

2020-07-26T03:00:40.4975849Z Gem::Ext::BuildError: ERROR: Failed to build gem native extension.
2020-07-26T03:00:40.4976473Z 
2020-07-26T03:00:40.4977046Z current directory:
2020-07-26T03:00:40.4978110Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/gsl-2.1.0.3/ext/gsl_native
2020-07-26T03:00:40.4978850Z /home/runner/.rubies/ruby-2.7.1/bin/ruby -I
2020-07-26T03:00:40.4979471Z /home/runner/.rubies/ruby-2.7.1/lib/ruby/2.7.0 -r
2020-07-26T03:00:40.4980071Z ./siteconf20200726-2689-qeqmg6.rb extconf.rb
2020-07-26T03:00:40.4980461Z *** ERROR: missing required library to compile this module: No such file or
2020-07-26T03:00:40.4980973Z directory - gsl-config
2020-07-26T03:00:40.4981343Z *** extconf.rb failed ***
2020-07-26T03:00:40.4981669Z Could not create Makefile due to some reason, probably lack of necessary
2020-07-26T03:00:40.4982013Z libraries and/or headers.  Check the mkmf.log file for more details.  You may
2020-07-26T03:00:40.4982324Z need configuration options.
2020-07-26T03:00:40.4982586Z 
2020-07-26T03:00:40.4982892Z Provided configuration options:
2020-07-26T03:00:40.4983385Z 	--with-opt-dir
2020-07-26T03:00:40.4983931Z 	--without-opt-dir
2020-07-26T03:00:40.4984471Z 	--with-opt-include
2020-07-26T03:00:40.4985036Z 	--without-opt-include=${opt-dir}/include
2020-07-26T03:00:40.4985572Z 	--with-opt-lib
2020-07-26T03:00:40.4986125Z 	--without-opt-lib=${opt-dir}/lib
2020-07-26T03:00:40.4986662Z 	--with-make-prog
2020-07-26T03:00:40.4987192Z 	--without-make-prog
2020-07-26T03:00:40.4987764Z 	--srcdir=.
2020-07-26T03:00:40.4988311Z 	--curdir
2020-07-26T03:00:40.4988896Z 	--ruby=/home/runner/.rubies/ruby-2.7.1/bin/$(RUBY_BASE_NAME)
2020-07-26T03:00:40.4989442Z 	--with-gsl-version
2020-07-26T03:00:40.4989740Z 
2020-07-26T03:00:40.4990056Z extconf failed, exit code 1
2020-07-26T03:00:40.4990316Z 
2020-07-26T03:00:40.4990620Z Gem files will remain installed in
2020-07-26T03:00:40.4991186Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/gsl-2.1.0.3
2020-07-26T03:00:40.4991576Z for inspection.
2020-07-26T03:00:40.4991888Z Results logged to
2020-07-26T03:00:40.4992519Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/extensions/x86_64-linux/2.7.0/gsl-2.1.0.3/gem_make.out
2020-07-26T03:00:40.4992877Z 
2020-07-26T03:00:40.4993259Z An error occurred while installing gsl (2.1.0.3), and Bundler cannot continue.
2020-07-26T03:00:40.4994249Z Make sure that `gem install gsl -v '2.1.0.3' --source 'https://rubygems.org/'`
2020-07-26T03:00:40.4994949Z succeeds before bundling.
2020-07-26T03:00:40.4995256Z 
2020-07-26T03:00:40.4995583Z In Gemfile:
2020-07-26T03:00:40.4995896Z   gsl

Apparently we can install ubuntu packages as part of the workflow though, so if we install the gsl libraries, this probably should work:

- name: install system dependencies
  run: sudo apt-get install libgsl-dev

0xdevalias · 2020-07-26T03:17:43Z

This build shows that installing the libgsl-dev system dependency works, but appears to take ~10sec to complete, at least on the first run. If this remains the same on subsequent builds, we may need to consider what the cost/benefit of this extra timing is, and/or alternatives we could use (eg. caching, prebuilt docker container, etc)

We also need to consider the install order, so that gsl knows to use nmatrix:

  Post-install message from gsl:
  
      gsl can be installed with or without narray support. Please install
      narray before and reinstall gsl if it is missing.
  
      gsl is also now compatible with NMatrix. Please install nmatrix before
      installing gsl.

The rb-gsl installation notes claim that we can forcibly tell it to use nmatrix:

In order to use rb-gsl with NMatrix you must first set the NMATRIX environment variable and then install rb-gsl:
gem install nmatrix export NMATRIX=1 gem install rb-gsl

This will compile rb-gsl with NMatrix specific functions.

As currently setup, even using --lsi, it looks as though the site only took ~9sec to generate.. would be interesting to see how that changes if we remove gsl and/or nmatrix:

jekyll build
  /home/runner/.rubies/ruby-2.7.1/bin/bundle exec jekyll build -s /home/runner/work/devalias.net/devalias.net/ --lsi
  Configuration file: /home/runner/work/devalias.net/devalias.net/_config.yml
              Source: /home/runner/work/devalias.net/devalias.net
         Destination: /home/runner/work/devalias.net/devalias.net/_site
   Incremental build: disabled. Enable with --incremental
        Generating... 
     GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
         Jekyll Feed: Generating feed for posts
                      done in 8.475 seconds.
   Auto-regeneration: disabled. Use --watch to enable.
  Took   9.13 seconds

To fix the follow warning, we need to set the appropriate environment variables for the Jekyll GitHub Metadata plugin:

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

https://github.com/jekyll/github-metadata/blob/master/docs/authentication.md
- JEKYLL_GITHUB_TOKEN

Can we use ${{ secrets.GITHUB_TOKEN }} for this? (no, see #86 (comment))

0xdevalias · 2020-07-26T03:35:35Z

Even when we removed nmatrix, our site build still seemed to take only ~9sec.

Unsure if this is because gsl wasn't actually using it before, or if it just has a negligible impact on our site build time.

Also, by using the cached gems, rather than having to re-install them, our total site dependencies/build step went from ~4min 51sec down to just ~16sec.

0xdevalias · 2020-07-26T03:40:46Z

Even when we removed gsl, our site build still seemed to take only ~8sec.

Unsure if this is because gsl just has a negligible impact on our site build time.

0xdevalias · 2020-07-26T04:00:09Z

Even when we removed Jekyll's --lsi option, the site build still only seemed to take ~7sec.

So maybe --lsi either isn't working, or just doesn't really have a big impact on our site build time.

In light of this.. i'm thinking we can leave the --lsi option enabled for now, but can probably remove the gsl/nmatrix optimisations we had added. Though we should probably do this in a follow up PR (see #87)

This also most likely renders #83 as irrelevant

(Edit: as was pointed out on #83, I may have disabled related_posts on my site a while ago, which is likely why the changes here weren't doing anything. Theres a lot of good info in #83 and linked to from #87 as well.. so worth exploring that more deeply before just removing it all)

0xdevalias · 2020-07-26T04:29:30Z

The build using secrets.GITHUB_TOKEN for JEKYLL_GITHUB_TOKEN failed with the following error during the site build:

2020-07-26T04:19:36.2417867Z �[31m  Liquid Exception: GET https://api.github.com/gists/1395926: 403 - Resource not accessible by integration // See: https://developer.github.com/v3/gists/#get-a-single-gist in /home/runner/work/devalias.net/devalias.net/_posts/2013-06-17-gists-on-tumblr.md�[0m
2020-07-26T04:19:36.2418778Z �[31m                    ------------------------------------------------�[0m
2020-07-26T04:19:36.2419520Z �[31m      Jekyll 4.1.1   Please append `--trace` to the `build` command �[0m
2020-07-26T04:19:36.2420234Z �[31m                     for any additional information or backtrace. �[0m
2020-07-26T04:19:36.2420916Z �[31m                    ------------------------------------------------�[0m
2020-07-26T04:19:36.2422515Z /home/runner/work/devalias.net/devalias.net/vendor/bundle/ruby/2.7.0/gems/octokit-4.18.0/lib/octokit/response/raise_error.rb:16:in `on_complete': GET https://api.github.com/gists/1395926: 403 - Resource not accessible by integration // See: https://developer.github.com/v3/gists/#get-a-single-gist (Octokit::Forbidden)

This appears to be caused by jekyll-gist:

https://github.com/jekyll/jekyll-gist
- Pro-tip: If you provide a personal access token with Gist scope, as the environmental variable JEKYLL_GITHUB_TOKEN, Jekyll Gist will use the Gist API to speed up site generation.

This same env var will also be used by the jekyll-github-metadata plugin, as per:

https://github.com/jekyll/github-metadata/blob/master/docs/authentication.md

Looking at the token I currently use for this on my local machine, it seems to only have the repo.public_repo scope enabled.

So perhaps creating a similar token, and configuring it as a GitHub Action secret that we then pass to this env var would work:

eg. ${{ secrets.JEKYLL_GITHUB_TOKEN_PAT }}

Edit: Thoughts from 2023 now that I know more about GitHub actions.. is there any reason why I can't just use the action token for this?

0xdevalias · 2020-07-26T05:37:43Z

When we added the deploy step to a test branch, the build seemed to complete successfully, taking only ~6sec to deploy to the test gh-pages branch.

The commit message was just deploy: 41e8aad, which differs from our previous message style of Update site: Wed 22 Jul 2020 15:19:29 AEST

actions-gh-pages lets us customise this using the commit_message or full_commit_message arguments, though i'm currently unsure how we would generate a similar formatted date to pass to it:

https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-custom-commit-message

There appear to be some actions related to getting the current time, unsure if there is a simpler way:

https://github.com/gerred/current-time
- uses core.setOutput("time", time)
https://github.com/srfrnk/current-time
- uses core.setOutput("time", time)
- https://github.com/1466587594/get-current-time
  - uses core.setOutput("formattedTime", moment().utcOffset(utcOffset).format(format));
https://github.com/ajilraju/actions-date
- appears to use coreutil's date command in an alpine docker container

Since most of these just seem to use core.setOutput, we may be able to do similar by using the workflow command echo syntax, and running directly on the host runner:

https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions
- echo "::workflow-command parameter1={data},parameter2={data}::{command value}"
https://docs.github.com/en/actions/reference/workflow-commands-for-github-actions#setting-an-output-parameter
- echo "::set-output name=action_fruit::strawberry"
https://docs.github.com/en/actions/creating-actions/metadata-syntax-for-github-actions#outputs
- The <output_id> must start with a letter or _ and contain only alphanumeric characters, -, or _.
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjobs_idoutputs
- Job outputs are available to all downstream jobs that depend on this job.
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepsrun
- Runs command-line programs using the operating system's shell.
https://docs.github.com/en/actions/reference/workflow-syntax-for-github-actions#jobsjob_idstepswith
- A map of the input parameters defined by the action. Each input parameter is a key/value pair. Input parameters are set as environment variables. The variable is prefixed with INPUT_ and converted to upper case.

Maybe something like:

- id: currentdate
  name: Get the current date and export it for later steps to use
  run: echo "::set-output name=date_str::$(date)"
  #run: echo "::set-output name=date_str::`date`"

0xdevalias · 2020-07-26T06:43:57Z

The build testing whether we can get a formatted date as step output (without using external actions) seemed to work:

      - id: currentdate
        name: 📅 get current date and time
        run: echo "::set-output name=date_str::$(date)"

      - name: 📅 check if date output was set correctly
        run: echo ${{steps.currentdate.outputs.date_str}}

Therefore, we can use them to customise the commit message for the generated site builds as per #86 (comment):

actions-gh-pages lets us customise this using the commit_message or full_commit_message arguments, though i'm currently unsure how we would generate a similar formatted date to pass to it:

https://github.com/peaceiris/actions-gh-pages#%EF%B8%8F-set-custom-commit-message

I think we want to use something like commit_message: ${{ steps.currentdate.outputs.date_str }}

0xdevalias

Changes required before this PR can be merged.

0xdevalias · 2020-07-26T06:58:42Z

.github/workflows/main.yml

+ push:
+ branches:
+ - master
+ - jekyll-build-deploy-workflow


This needs to be removed before this PR is merged, and/or a better solution for testing in non-master PR's implemented (that won't override our production site deployment)

0xdevalias · 2020-07-26T06:59:07Z

.github/workflows/main.yml

+ - jekyll-build-deploy-workflow
+ paths-ignore:
+ - .cache/**
+ # - .github/**


uncomment this?

0xdevalias · 2020-07-26T06:59:42Z

.github/workflows/main.yml

+ - name: 🔨 install system dependencies
+ run: sudo apt-get install libgsl-dev


this can be cleaned up as part of #87

If we even want to clean it up anymore.. see #87 and #83 for more context/details.

0xdevalias · 2020-07-26T07:00:11Z

.github/workflows/main.yml

+ uses: limjh16/jekyll-action-ts@v2
+ with:
+ enable_cache: true
+# format_output: true


Do we want to enable this? Otherwise, remove it.

0xdevalias · 2020-07-26T07:00:42Z

.github/workflows/main.yml

+ with:
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ publish_dir: ./_site
+ publish_branch: jekyll-build-deploy-workflow-gh-pages


This needs to be changed to the proper gh-pages branch before this PR is merged, and/or a better solution for testing in non-master PR's implemented (that won't override our production site deployment)

initial jekyll build/deploy workflow

a9390f7

0xdevalias self-assigned this Jul 26, 2020

allow workflow to trigger from this branch for testing purposes

7a78104

install system dependencies required to compile native gems

dc2a10f

0xdevalias linked an issue Jul 26, 2020 that may be closed by this pull request

Dockerize Jekyll + automate site builds #27

Open

try removing nmatrix

6f229a6

try removing gsl

0962d23

add jekyll --profile arg

ce195f0

0xdevalias mentioned this pull request Jul 26, 2020

Speed up jekyll related posts functionality (--lsi, classifier-reborn, gsl, nmatrix, narray, Numo:NArray, Numo:GSL) #83

Open

0xdevalias added 2 commits July 26, 2020 14:14

try without jekyll --lsi arg

ba396ed

restore --lsi option + gsl/nmatrix gems

e05b0b2

0xdevalias force-pushed the jekyll-build-deploy-workflow branch from f7406f5 to e05b0b2 Compare July 26, 2020 04:14

add JEKYLL_GITHUB_TOKEN env

0f2c882

0xdevalias added 2 commits July 26, 2020 14:35

use JEKYLL_GITHUB_TOKEN_PAT secret in JEKYLL_GITHUB_TOKEN env

fbd61d7

add gh-pages deploy step to workflow with test branch

41e8aad

try storing current date in an output

9e854b2

0xdevalias added 2 commits July 26, 2020 16:46

use current date output in site deploy commit message

68b59e9

improve site deploy commit message

605ee6b

0xdevalias mentioned this pull request Jul 26, 2020

Remove gsl/nmatrix dependencies from Gemfile and build workflow #87

Open

0xdevalias commented Jul 26, 2020

View reviewed changes

0xdevalias mentioned this pull request Jun 20, 2024

Dockerize Jekyll + automate site builds #27

Open

0xdevalias mentioned this pull request Jun 20, 2024

Upgrade from Jekyll 4.1.0 to latest (4.3.3+) #111

Open

0xdevalias mentioned this pull request Jul 3, 2024

disable/remove jekyll related posts (--lsi) + related dependencies #114

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Jekyll build/deploy workflow #86

Jekyll build/deploy workflow #86

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020

0xdevalias left a comment

0xdevalias Jul 26, 2020

0xdevalias Jul 26, 2020

0xdevalias Jul 26, 2020

0xdevalias Jan 30, 2023

0xdevalias Jul 26, 2020

0xdevalias Jul 26, 2020

		- name: 🔨 install system dependencies
		run: sudo apt-get install libgsl-dev

Jekyll build/deploy workflow #86

Are you sure you want to change the base?

Jekyll build/deploy workflow #86

Conversation

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020 • edited Loading

0xdevalias commented Jul 26, 2020

0xdevalias left a comment

Choose a reason for hiding this comment

0xdevalias Jul 26, 2020

Choose a reason for hiding this comment

0xdevalias Jul 26, 2020

Choose a reason for hiding this comment

0xdevalias Jul 26, 2020

Choose a reason for hiding this comment

0xdevalias Jan 30, 2023

Choose a reason for hiding this comment

0xdevalias Jul 26, 2020

Choose a reason for hiding this comment

0xdevalias Jul 26, 2020

Choose a reason for hiding this comment

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading

0xdevalias commented Jul 26, 2020 •

edited

Loading