Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dockerize Jekyll + automate site builds #27

Open
0xdevalias opened this issue Dec 19, 2019 · 2 comments · May be fixed by #86
Open

Dockerize Jekyll + automate site builds #27

0xdevalias opened this issue Dec 19, 2019 · 2 comments · May be fixed by #86
Assignees
@0xdevalias

Comments

@0xdevalias
Copy link
Owner

0xdevalias commented Dec 19, 2019 •
edited
Loading

Dockerize the blog + setup automated site build/deploy on code push

Refs

According to github/pages-gem#651, the best recommended way to support Jekyll 4 on GitHub pages currently is to use GitHub actions:

See Also
@0xdevalias 0xdevalias self-assigned this Jul 22, 2020
@0xdevalias 0xdevalias linked a pull request Jul 26, 2020 that will close this issue
Jekyll build/deploy workflow #86
Open
@0xdevalias
Copy link
Owner Author

0xdevalias commented Jul 28, 2022 •
edited
Loading

tl;dr:

👋 !

We just announced official support for building/deploying Pages straight from GitHub Actions: https://github.blog/changelog/2022-07-27-github-pages-custom-github-actions-workflows-beta/

No hacks (no .nojekyll, no CNAME files), no need to push content back to a dedicated branch, etc. It's "streamlined" and while it doesn't look like much on the surface, lots of things have been simplified underneath to make it happen.

We wrote starter workflows for Pages for a few static site generators including one for plain Jekyll. By that I mean, a Gemfile is required and so is a _config.yml file but there are no "vendor locks" (in the name of security): you can depend on the latest version of Jekyll, use all plugins and all the themes you want. We don't meddle with your configuration file either because Actions is a secure sandbox.

I am aware that this is not a drop in replacement for what pages-gem provides today (a default theme, configuration file, etc.). This is also not exactly the objective of this new feature but we have to start somewhere! pages-gem today fills two goals: (a) provide the security requirements for a non-Actions build infrastructure, (b) provide default so sites can be built out of markdown files seamlessly without a user needing to even know what Jekyll is in the first place. I am not sure yet that we need pages-gem to answer (b) in a GitHub Actions world. I am hopefully my team or the community will figure out something soon enough.

This whole feature is in public beta today and I am very excited about it 🥳 This is certainly not perfect and will be iterated on but we want to hear your feedback. Here or on the new community site.

Originally posted by @yoannchaudet in github/pages-gem#651 (comment)

@0xdevalias
Copy link
Owner Author

0xdevalias commented Jun 20, 2024 •
edited
Loading

Notes for doing a local build + deploy currently

Apparently I already started upgrading things back in ~2020.. which I'd completely forgotten about; so there may be useful notes/snippets of long lost wisdom there:

There's also a newer 'update jekyll' issue here, to bring things up to date again:

bundle install (on ruby 3.1.1)

Looks like we can't use ruby 3.1.1 (an arbitrary 3.x version I had installed) without updating some of the dependencies:

⇒ rbenv local 3.1.1

⇒ bundle install
Bundler 2.3.9 is running, but your lockfile was generated with 2.1.4. Installing Bundler 2.1.4 and restarting using that version.
Fetching gem metadata from https://rubygems.org/.
Fetching bundler 2.1.4
Installing bundler 2.1.4
Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated. Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
Fetching gem metadata from https://rubygems.org/.........
listen-3.2.1 requires ruby version >= 2.2.7, ~> 2.2, which is incompatible with the current version, ruby 3.1.1p18

rbenv install ruby 2.7.1

So let's try with the version in the .ruby-version file:

rbenv install ruby 2.7.1

But that errored during compilation:

..snip..

ossl_pkey_dsa.c:394:11: error: incompatible function pointer types assigning to 'int (*)(DSA *, unsigned char **)' (aka 'int (*)(struct dsa_st *, unsigned char **)') from 'int (const DSA *, unsigned char **)' (aka 'int (const struct dsa_st *, unsigned char **)') [-Wincompatible-function-pointer-types]
        i2d_func = i2d_DSA_PUBKEY;
                 ^ ~~~~~~~~~~~~~~

..snip..

78 warnings and 1 error generated.
make[2]: *** [ossl_pkey_dsa.o] Error 1
make[1]: *** [ext/openssl/all] Error 2
make: *** [build-ext] Error 2
!!! Compiling ruby 2.7.1 failed!

Googling for some bits of that error led me to this:

Which led me to looking back at my build logs and seeing this part:

..snip..

[email protected] is keg-only, which means it was not symlinked into /usr/local,
because this is an alternate version of another formula.

If you need to have [email protected] first in your PATH, run:
  echo 'export PATH="/usr/local/opt/[email protected]/bin:$PATH"' >> ~/.zshrc

For compilers to find [email protected] you may need to set:
  export LDFLAGS="-L/usr/local/opt/[email protected]/lib"
  export CPPFLAGS="-I/usr/local/opt/[email protected]/include"

For pkg-config to find [email protected] you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig"

..snip..

All those env flags except for PATH were empty in my shell, so I tried this:

export PATH="/usr/local/opt/[email protected]/bin:$PATH";
export LDFLAGS="-L/usr/local/opt/[email protected]/lib";
export CPPFLAGS="-I/usr/local/opt/[email protected]/include";
export PKG_CONFIG_PATH="/usr/local/opt/[email protected]/lib/pkgconfig";

Then tried the install again, which succeeded ✅:

⇒ rbenv install ruby 2.7.1
Note: This is just a small wrapper around 'ruby-install' to prefill the 'install-dir' params for rbenv support

It looks like you're trying to install
  flavor  : ruby
  version : 2.7.1

We will install with the following command:
  ruby-install  --install-dir '/Users/devalias/.rbenv/versions/2.7.1' 'ruby' '2.7.1'

Is that ok? [Y/n]: Y

..snip..

>>> Successfully installed ruby 2.7.1 into /Users/devalias/.rbenv/versions/2.7.1

If we wanted to, we could then cleanup the env vars again with something like:

export PATH=${PATH#/usr/local/opt/openssl@1.1/bin:}
unset LDFLAGS
unset CPPFLAGS
unset PKG_CONFIG_PATH

Or just restart our shell to get the defaults back again.

bundle install (on ruby 2.7.1)

Ensuring we are still using ruby 2.7.1 as per the .ruby-version file:

⇒ rbenv local 2.7.1

We try bundle install again:

⇒ bundle install

..snip..

Gem::Ext::BuildError: ERROR: Failed to build gem native extension.

    current directory: /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/gsl-2.1.0.3/ext/gsl_native
/Users/devalias/.rbenv/versions/2.7.1/bin/ruby -I /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/2.7.0 -r ./siteconf20240620-69761-1fcewq5.rb extconf.rb
*** ERROR: missing required library to compile this module: No such file or directory - gsl-config
*** extconf.rb failed ***
Could not create Makefile due to some reason, probably lack of necessary
libraries and/or headers.  Check the mkmf.log file for more details.  You may
need configuration options.

Provided configuration options:
	--with-opt-dir
	--with-opt-include
	--without-opt-include=${opt-dir}/include
	--with-opt-lib
	--without-opt-lib=${opt-dir}/lib
	--with-make-prog
	--without-make-prog
	--srcdir=.
	--curdir
	--ruby=/Users/devalias/.rbenv/versions/2.7.1/bin/$(RUBY_BASE_NAME)
	--with-gsl-version

extconf failed, exit code 1

Gem files will remain installed in /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/gems/gsl-2.1.0.3 for inspection.
Results logged to /Users/devalias/.rbenv/versions/2.7.1/lib/ruby/gems/2.7.0/extensions/x86_64-darwin-23/2.7.0-static/gsl-2.1.0.3/gem_make.out

An error occurred while installing gsl (2.1.0.3), and Bundler cannot continue.
Make sure that `gem install gsl -v '2.1.0.3' --source 'https://rubygems.org/'` succeeds before bundling.

In Gemfile:
  gsl

But get an error while installing gsl:

ERROR: missing required library to compile this module: No such file or directory - gsl-config

Which it seems we have seen/resolved before:

But this time seem to be hitting new issues with it:

These issues might be relevant here:

Since it sounds like we're not even using the site.related_posts feature that --lsi / gsl / etc are meant to be speeding up (Ref); there's probably no harm in just commenting out those gems for now, at least as a temporary hack to get things working:

  # ..snip..
- gem 'nmatrix'
- gem 'gsl' # Note: you need to install a compatible version (eg. 2.1) of gsl first: brew install [email protected]
- gem 'classifier-reborn'
+ # gem 'nmatrix'
+ # gem 'gsl' # Note: you need to install a compatible version (eg. 2.1) of gsl first: brew install [email protected]
+ # gem 'classifier-reborn'
  # ..snip..

Which then allows us to complete our bundle install:

⇒ bundle install
Fetching gem metadata from https://rubygems.org/.........

..snip..

Bundle complete! 15 Gemfile dependencies, 48 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.

The following PR commits these changes more permanently (at least for now):

Building the site

Now to remember how to build/deploy the site.. which seems to be scattered across a number of half documented places:

In my 'upgrade to Jekyll 4.x' PR, I seem to have changed the suggested commands in PUBLISHING.md for Build/Deploy to use ./bin/build and ./bin/deploy.. so I guess they're probably the main ones I should be using now:

Running bin/build:

⇒ ./bin/build
Configuration file: /Users/devalias/dev/0xdevalias/devalias.net/_config.yml
            Source: /Users/devalias/dev/0xdevalias/devalias.net
       Destination: /Users/devalias/dev/0xdevalias/devalias.net/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
   GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.
       Jekyll Feed: Generating feed for posts

Build Process Summary:

| PHASE      |    TIME |
+------------+---------+
| RESET      |  7.8864 |
| READ       |  0.1378 |
| GENERATE   |  0.4316 |
| RENDER     | 22.9618 |
| CLEANUP    |  0.0262 |
| WRITE      |  1.1139 |
+------------+---------+
| TOTAL TIME | 32.5577 |


Site Render Stats:

| Filename                                                                   | Count |     Bytes |   Time |
+----------------------------------------------------------------------------+-------+-----------+--------+
| _layouts/default.html                                                      |   647 |  8684.62K | 16.305 |
| _includes/head/head.html                                                   |   647 |  2574.81K |  1.860 |
| _includes/loop.html                                                        |   537 |  1937.34K |  1.308 |
| _includes/post-meta.html                                                   |   931 |  1061.73K |  0.719 |
| _includes/navigation.html                                                  |   647 |  1009.04K |  0.381 |
| _posts/2013-06-17-gists-on-tumblr.md                                       |     1 |     0.27K |  0.361 |
| _posts/2013-07-13-nmap-sh-saving-precious-seconds.md                       |     1 |     0.52K |  0.358 |
| _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md       |     1 |     0.48K |  0.353 |
| _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md               |     1 |     0.54K |  0.348 |
| _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md                   |     1 |     1.03K |  0.347 |
| _posts/2015-04-20-dogedraw-now-with-more-nyaan.md                          |     1 |     0.79K |  0.345 |
| _posts/2014-05-15-java-scala-future-promise-map-headsplode.md              |     1 |     0.50K |  0.344 |
| _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md |     1 |     0.91K |  0.328 |
| _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md  |     1 |     0.86K |  0.320 |
| _layouts/post.html                                                         |    77 |  1159.14K |  0.253 |
| _includes/header.html                                                      |   647 |   563.42K |  0.160 |
| sitemap.xml                                                                |     1 |    47.27K |  0.103 |
| _includes/tag_pagination.html                                              |  1010 |   220.11K |  0.084 |
| feed.xml                                                                   |     1 |   167.83K |  0.075 |
| _layouts/atom.xml                                                          |   494 |  5956.01K |  0.062 |
| _includes/author-social-icons.html                                         |    93 |   261.40K |  0.046 |
| _includes/head/styles.html                                                 |   647 |   236.31K |  0.033 |
| _includes/scripts/analytics.html                                           |   648 |   202.50K |  0.024 |
| tag/hypernova/index.html                                                   |     1 |     4.96K |  0.020 |
| 3/index.html                                                               |     1 |    10.80K |  0.020 |
| tag/bootzooka/index.html                                                   |     1 |     4.96K |  0.016 |
| _includes/social-share.html                                                |    77 |   116.78K |  0.016 |
| tag/greatfet/index.html                                                    |     1 |     7.81K |  0.015 |
| tag/redux/index.html                                                       |     1 |     3.76K |  0.014 |
| tag/cashay/index.html                                                      |     1 |     3.77K |  0.013 |
| tag/debug/index.html                                                       |     1 |     3.32K |  0.012 |
| _layouts/page.html                                                         |    33 |   123.26K |  0.012 |
| 5/index.html                                                               |     1 |    12.06K |  0.012 |
| _includes/author-meta.html                                                 |    93 |    37.62K |  0.011 |
| index.html                                                                 |     1 |    12.13K |  0.011 |
| tag/backend/index.html                                                     |     1 |     4.96K |  0.011 |
| 4/index.html                                                               |     1 |    11.29K |  0.010 |
| tag/dev/index.html                                                         |     1 |    12.71K |  0.010 |
| author/devalias/11/index.html                                              |     1 |    13.84K |  0.010 |
| tag/hack/index.html                                                        |     1 |    10.91K |  0.010 |
| tag/pentest/index.html                                                     |     1 |    10.98K |  0.010 |
| tag/laser/index.html                                                       |     1 |     3.59K |  0.010 |
| tag/akka-http/index.html                                                   |     1 |     7.05K |  0.010 |
| tag/babel/index.html                                                       |     1 |     4.95K |  0.009 |
| tag/coffeescript/index.html                                                |     1 |     4.97K |  0.009 |
| tag/amazon/index.html                                                      |     1 |     3.71K |  0.009 |
| tag/python/index.html                                                      |     1 |     6.58K |  0.009 |
| 9/index.html                                                               |     1 |    10.91K |  0.009 |
| tag/sources.list/index.html                                                |     1 |     3.31K |  0.009 |
| tag/usb/index.html                                                         |     1 |     6.38K |  0.009 |
+----------------------------------------------------------------------------+-------+-----------+--------+
| TOTAL (for 50 files)                                                       |  7263 | 24544.81K | 24.836 |

                    done in 32.599 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

There's a warning that seems to be from jekyll-github-metadata:

GitHub Metadata: No GitHub API authentication could be found. Some fields may be missing or have incorrect data.

It seems we need to configure an access token for it:

Which I have now configured, and added a placeholder for to my dotfiles:

Re-running bin/build:

⇒ ./bin/build
Configuration file: /Users/devalias/dev/0xdevalias/devalias.net/_config.yml
            Source: /Users/devalias/dev/0xdevalias/devalias.net
       Destination: /Users/devalias/dev/0xdevalias/devalias.net/_site
 Incremental build: disabled. Enable with --incremental
      Generating...
       Jekyll Feed: Generating feed for posts

Build Process Summary:

| PHASE      |    TIME |
+------------+---------+
| RESET      |  8.3194 |
| READ       |  0.1158 |
| GENERATE   |  0.4404 |
| RENDER     | 23.9001 |
| CLEANUP    |  0.1186 |
| WRITE      |  0.5646 |
+------------+---------+
| TOTAL TIME | 33.4589 |


Site Render Stats:

| Filename                                                                   | Count |     Bytes |   Time |
+----------------------------------------------------------------------------+-------+-----------+--------+
| _layouts/default.html                                                      |   647 |  8698.90K | 16.015 |
| _includes/head/head.html                                                   |   647 |  2574.81K |  1.592 |
| _includes/loop.html                                                        |   537 |  1936.73K |  1.187 |
| _posts/2013-06-17-gists-on-tumblr.md                                       |     1 |     0.82K |  1.001 |
| _includes/post-meta.html                                                   |   931 |  1061.07K |  0.655 |
| _posts/2014-05-15-java-scala-future-promise-map-headsplode.md              |     1 |     2.37K |  0.514 |
| _posts/2014-04-02-hacking-unicoins-for-fun-and-profit.md                   |     1 |     2.28K |  0.500 |
| _posts/2013-08-10-erpscan-automator-because-manual-is-meh.md               |     1 |     2.90K |  0.479 |
| _posts/2013-11-21-vfeed-wrapper-helper-scripts-for-speed-and-efficiency.md |     1 |     1.65K |  0.475 |
| _posts/2015-04-20-dogedraw-now-with-more-nyaan.md                          |     1 |     4.38K |  0.462 |
| _posts/2013-07-13-nmap-sh-saving-precious-seconds.md                       |     1 |     1.28K |  0.459 |
| _posts/2013-08-03-rails-lessons-learned-the-hard-way-1-db-migrate.md       |     1 |     0.53K |  0.431 |
| _posts/2013-08-29-reversing-powershell-securestring-for-fun-and-profit.md  |     1 |     2.24K |  0.427 |
| _includes/navigation.html                                                  |   647 |  1009.04K |  0.359 |
| _layouts/post.html                                                         |    77 |  1174.03K |  0.254 |
| _includes/header.html                                                      |   647 |   563.42K |  0.153 |
| sitemap.xml                                                                |     1 |    47.27K |  0.078 |
| _includes/tag_pagination.html                                              |  1010 |   220.11K |  0.077 |
| _layouts/atom.xml                                                          |   494 |  6078.07K |  0.061 |
| feed.xml                                                                   |     1 |   167.83K |  0.049 |
| _includes/author-social-icons.html                                         |    93 |   261.40K |  0.048 |
| _includes/head/styles.html                                                 |   647 |   236.31K |  0.032 |
| _includes/scripts/analytics.html                                           |   648 |   202.50K |  0.022 |
| tag/sectalks/rss.xml                                                       |     1 |    21.37K |  0.018 |
| _includes/social-share.html                                                |    77 |   116.78K |  0.015 |
| _layouts/page.html                                                         |    33 |   123.26K |  0.014 |
| 2/index.html                                                               |     1 |    10.69K |  0.014 |
| tag/kali/index.html                                                        |     1 |     5.65K |  0.013 |
| index.html                                                                 |     1 |    12.13K |  0.012 |
| tag/wireshark/index.html                                                   |     1 |     4.75K |  0.012 |
| _includes/author-meta.html                                                 |    93 |    37.62K |  0.011 |
| author/devalias/index.html                                                 |     1 |    15.47K |  0.011 |
| tag/bulletproof/index.html                                                 |     1 |    11.68K |  0.011 |
| tag/npm/index.html                                                         |     1 |     3.30K |  0.011 |
| tag/bash/index.html                                                        |     1 |     6.32K |  0.010 |
| tag/facebook/index.html                                                    |     1 |     6.34K |  0.009 |
| tag/bulletproof/2/index.html                                               |     1 |     8.87K |  0.009 |
| tag/dev/index.html                                                         |     1 |    12.71K |  0.009 |
| author/devalias/3/index.html                                               |     1 |    14.18K |  0.009 |
| _includes/author-image.html                                                |    93 |    18.62K |  0.008 |
| tag/problem/index.html                                                     |     1 |     3.16K |  0.008 |
| tag/bulletproof-quarterly/index.html                                       |     1 |    11.73K |  0.008 |
| tag/rest/index.html                                                        |     1 |     3.76K |  0.008 |
| _includes/head/indie-auth.html                                             |   647 |   171.23K |  0.008 |
| tag/babel/index.html                                                       |     1 |     4.95K |  0.008 |
| tag/apt-get/index.html                                                     |     1 |     5.66K |  0.007 |
| tag/gh-pages/index.html                                                    |     1 |     3.06K |  0.007 |
| tag/security/2/index.html                                                  |     1 |     9.64K |  0.007 |
| author/devalias/5/index.html                                               |     1 |    15.43K |  0.007 |
| tag/relay/index.html                                                       |     1 |     3.76K |  0.007 |
+----------------------------------------------------------------------------+-------+-----------+--------+
| TOTAL (for 50 files)                                                       |  8001 | 24912.03K | 25.601 |

                    done in 33.501 seconds.
 Auto-regeneration: disabled. Use --watch to enable.

That seemed to fix the warning! ✅

Deploying the site

Now that we've built the site, time to deploy it!

As we figured out earlier, bin/deploy is probably what we want to use here:

  • https://github.com/0xdevalias/devalias.net/blob/master/bin/deploy
    • https://github.com/vwochnik/jekyll-deploy

      • Jekyll Deploy is a Jekyll plugin which adds a deploy sub-command to the jekyll executable which allows deploy commands to be executed quickly.

      • The deploy command executes all commands specified in the deploy array inside the site's configuration file

        • devalias.net/_config.yml

          Lines 105 to 111 in 58ea83d

          # https://github.com/vwochnik/jekyll-deploy
          deploy:
          - touch .nojekyll
          - git add -A
          - 'git commit -m "Update site: `date`"'
          - git push origin gh-pages
          #- git subtree push --prefix _site origin gh-pages

      • All commands are executed individually within the site's destination directory which is, by default, _site but can be changed with the destination configuration option.

      • The deploy sub-command supports the built-in --config, --destination and --verbose command line options.

So it sounds like running bin/deploy will deploy the site we just built with bin/build (that ended up in ./_site) to GitHub pages.. let's find out!

From memory, the way we used to deploy this is that _site was actually a separately cloned git repository mapped to the gh-pages branch of the repo (which was originally made as a disconnected branch so it doesn't share a parent commit with any of the 'source code' for the site); but looking at it currently on this local machine, there doesn't seem to be a separate _site/.git:

⇒ ls -la _site/.git
gls: cannot access '_site/.git': No such file or directory

We can see that we have it .gitignore'd for the main branch:

devalias.net/.gitignore

Lines 4 to 8 in 58ea83d

# Jekyll build
_site
.jekyll-metadata
.sass-cache
.jekyll-cache/Jekyll

And there is seemingly no .gitignore on the gh-pages branch; which is how we would expect it to be:

https://github.com/0xdevalias/devalias.net/blob/gh-pages/.gitignore

Based on that.. I would kind of expect bin/deploy to fail currently.. or at the very least, to do things we're sort of not expecting, since the git add -A would add to the main repo, not the other clone.. Though since we have all the contents of that folder .gitignore'd, I suspect it may just end up trying to make an empty commit, and then probably failing..

I dug into things a bit, and refactored bin/build and bin/deploy to be more robust to ensuring the _site git branch is cloned/setup properly; as well as not failing if there are no commits to add, etc.

Following those instructions, we got a deploy to work:

Which we can see the deploy run for here:

We did some manual checks, and everything looked good, so were able to finalise that PR/build/deployment:

This was referenced Jun 20, 2024
Open
Open
Merged
Open
@0xdevalias 0xdevalias mentioned this issue Jul 3, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@0xdevalias 0xdevalias

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Jekyll build/deploy workflow 0xdevalias/devalias.net
1 participant
@0xdevalias