Can Eleventy cache filters or template includes? (memoize) #840

KyleMit · 2020-01-03T01:00:22Z

I'm using both the Quick Tips here:

So my default layout looks roughly like this:

<!doctype html>
<html lang="en">
<head>

    <title>{{title}}</title>
    
    {% set css %}
        {% include "assets/styles/fonts.css" %}
        {% include "assets/styles/styles.css" %}
        {% include "assets/styles/hljs.css" %}
    {% endset %}
    <style type="text/css">{{ css | cssmin | safe }}</style>

</head>
<body >

    <main class="content">
        {{ content | safe }}
    </main>

    {% set js %}
        {% include "assets/scripts/polyfill.js" %}
        {% include "assets/scripts/nav-toggle.js" %}
        {% include "assets/scripts/offline.js" %}
        {% include "assets/scripts/copy-snippet.js" %}
        {% include "assets/scripts/tooltip.js" %}
        {% include "assets/scripts/site.js" %}
    {% endset %}
    <script type="text/javascript">{{ js | jsmin | safe }}</script>

</body>
</html>

Which works to embed any sitewide CSS & JS inline on every single page

Every page will be the same (and if I do need page-specific styles, I can handle them separately) - so it would be preferable for build times to only do the JS and CSS minification once, since the input should be identical and the minification should be a pure function

Currently, the build times scale linearly with the number of pages being built and the filters take up about 10 seconds out of a 12.5s build:

And a simple log on the filter function confirms that the same process is executed on every single page build.

Perhaps I can memoize the minifiy function, but since it's processing about 20k characters, the content hash is gonna be huge.

Another idea was to pull out the scripts into a separate template:

File: /_includes/partials/scripts.njk

{% set js %}
    {% include "assets/scripts/polyfill.js" %}
    {% include "assets/scripts/nav-toggle.js" %}
    {% include "assets/scripts/offline.js" %}
    {% include "assets/scripts/copy-snippet.js" %}
    {% include "assets/scripts/tooltip.js" %}
    {% include "assets/scripts/site.js" %}
{% endset %}
<script type="text/javascript">{{ js | jsmin | safe }}</script>

And then include it in my layout page:

File: /layouts/default.njk

{% include "_partials/scripts.njk" %}

But that would be under the hopes that there's a way to cache a template include?

The text was updated successfully, but these errors were encountered:

KyleMit · 2020-01-03T03:47:45Z

One place that does cache output evaluation is in processing templates. So we can add the .njk file as content, instead of adding the scripts as a partial template in the includes folder.

I already have a meta folder for content that needs to be processed. Meta isn't sacred, but it's a nice place to put site stuff without cluttering more contentful pages - so we can add it there:

2019.vtcodecamp
├── _includes/              # runtime includes
│   ├── scripts/            # script includes
│   │   ├── polyfill.js
│   │   ├── nav-toggle.js
│   │   └── site.js
├── _layout/                # layout templates
│   └── default.njk
├── posts/                  # content posts
├── pages/                  # core pages
├── meta/                   # site meta content
│   ├── search.11ty.js      # /search.json
│   ├── rss-feed.njk        # /feed.xml
│   └── script-bundle.njk   # /scripts.js
└── .eleventy.js            # config information for 11ty

Option A) - Add to Includes

If we use eleventyExcludeFromCollections & permalinkBypassOutputDir, we can add the bundled output alongside the other includes

File: /meta/script-bundle.njk

---
eleventyExcludeFromCollections: true
permalink: "_includes/scripts/bundle.js"
permalinkBypassOutputDir: true
---

{% set js %}
    {% include "_includes/scripts/polyfill.js" %}
    {% include "_includes/scripts/nav-toggle.js" %}
    {% include "_includes/scripts/site.js" %}
{% endset %}
{{ js | jsmin | safe }}

It's important for both git and the eleventy watch process that you not commit this file since it's technically a build artifact - so update your .gitignore as well

File: /.gitignore

_includes/scripts/bundle.js

Then the default layout can consume the bundle we just built as if it were regular includes

File: /_layout/default.njk

<script type="text/javascript">{% include "_includes/scripts/bundle.js" %}</script>

CAUTION: This works almost perfectly, however seems to fail on the first build where the bundle.js is always n-1 away from the current build. Even though the bundler template seem to be prioritized first, the output isn't loaded and available to the rest of the templates for the rest of the build

Option B) - Add to Collections

If we didn't prevent it being added to collections as in option a, we could read the templateContent property from collections.

Here's what our script bundler would look like:

File: /meta/script-bundle.njk

---
permalink: "/scripts/bundle.js"
---

{% set js %}
    {% include "_includes/scripts/polyfill.js" %}
    {% include "_includes/scripts/nav-toggle.js" %}
    {% include "_includes/scripts/site.js" %}
{% endset %}
{{ js | jsmin | safe }}

That can be injected into any template via {{collections[i].templateContent}} - but we have to find which position the script bundle is in the collection, so it probably makes sense to use eleventyConfig.addCollection to figure out using getFilteredByGlob or getFilteredByTag if you want to add a unique tag

File: .eleventy.js

eleventyConfig.addCollection("bundles", col => {
    let scriptCol = col.getFilteredByGlob("**/meta/script-bundle.njk")
    return {
        script: scriptCol[0]
    }
});

And then access the collection from the layout template like this:

File: /_layout/default.njk

<script type="text/javascript">
    {{ collections.bundles.script.templateContent | safe }}
</script>

And here's our new and improved metrics for a 75% reduction in build times and 90% reduction in filter load:

zachleat · 2024-03-26T14:04:50Z

via https://chrisburnell.com/article/memoizing-asset-bundles/

zachleat · 2024-07-02T19:30:20Z

3.0.0-alpha.15 will ship with a memoization layer around the slug, slugify, and inputPathToUrl filters.

Note that benchmarking output will still reflect the unmemoized call count (e.g. 998× below):

  Eleventy:Benchmark Benchmark      4ms   0%   998× (Configuration) "slug" Universal Filter +1ms

Note that you can memoize a filter or shortcode in your Eleventy configuration file (in any version) right now. Here’s an example using the memoize package https://www.npmjs.com/package/memoize:

import memoize from "memoize";

export default function(eleventyConfig) {
	eleventyConfig.addLiquidFilter("htmlEntities", memoize(str => {
		return encode(str);
	}));
};

I think this usage is probably the right level of abstraction for most use cases, rather than complicating the configuration API with extra stuff.

KyleMit added the education label Jan 3, 2020

KyleMit mentioned this issue Jan 3, 2020

Fix Build Perf for CSS & JS Minification VermontDepartmentOfHealth/docgov#87

Closed

nhoizey mentioned this issue Jan 12, 2020

Optimize build performance nhoizey/nicolas-hoizey.com#173

Closed

nhoizey mentioned this issue Mar 12, 2023

Would it make sense to memoize some functions in Eleventy core? #2861

Closed

zachleat added enhancement and removed education labels Mar 26, 2024

zachleat changed the title ~~Can Eleventy cache filters or template includes?~~ Can Eleventy cache filters or template includes? (memoize) Mar 26, 2024

zachleat added the enhancement: favorite Vanity label! The maintainer likes this enhancement request a lot. label Mar 26, 2024

zachleat added a commit that referenced this issue Jul 2, 2024

Memoize the slug, slugify, and inputPathToUrl filters. #840

a9fe852

zachleat added this to the Eleventy 3.0.0 milestone Jul 2, 2024

zachleat added the education label Jul 2, 2024

zachleat closed this as completed Jul 2, 2024

zachleat added the needs-documentation Documentation for this issue/feature is pending! label Jul 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can Eleventy cache filters or template includes? (memoize) #840

Can Eleventy cache filters or template includes? (memoize) #840

KyleMit commented Jan 3, 2020

KyleMit commented Jan 3, 2020

zachleat commented Mar 26, 2024

zachleat commented Jul 2, 2024 •

edited

Loading

Can Eleventy cache filters or template includes? (memoize) #840

Can Eleventy cache filters or template includes? (memoize) #840

Comments

KyleMit commented Jan 3, 2020

KyleMit commented Jan 3, 2020

Option A) - Add to Includes

Option B) - Add to Collections

zachleat commented Mar 26, 2024

zachleat commented Jul 2, 2024 • edited Loading

zachleat commented Jul 2, 2024 •

edited

Loading