Collections in Javascript templates

[gerwitz]

I am using an 11ty.js template to generate a search index.

The data method looks like this:

  data() {
    return {
      permalink: "/search/lunr.json",
      eleventyExcludeFromCollections: "true"
    };
  }

I know eleventyExcludeFromCollections value is working, in that a sitemap (.njk) template that uses collections.all does not include this file when it is present.

But even still, attempting to use collections.all from within my render() throws a UsingCircularTemplateContentReferenceError.

Using a more narrowly-scoped collection is fine. I know I can define an "everything except the search index" collection for this, but shouldn't I be able to just use .all?

[Ryuno-Ki]

Hi @gerwitz,
from where are you calling this method?
I am querying the GitHub API from a Global Data File named _data/github.js (thus, creating a collection „github”).

How do you consume the collection?
Could you share some template snippet?

[gerwitz]

This is not a data file, but rather a Javascript template. The version below is working, but if I replace collections.writing with collections.all I get the aforementioned error.

class LunrIndex {
  data() {
    return {
      permalink: "/search/lunr.json",
      eleventyExcludeFromCollections: true
    };
  }

  render(data) {
    var pages = data.collections.writing.map(function(page) {
      return {
        url: page.url,
        title: page.data.title,
        date: page.date,
        content: page.templateContent
      };
    });
    return JSON.stringify(pages);
  }
}

module.exports = LunrIndex;

[Ryuno-Ki]

Hm, I still don't have enough information to reproduce it locally.
Here's what I have so far:

.
├── index.md
├── package.json
├── package-lock.json
├── search
│   ├── index.md
│   └── lunr.11ty.js
└── writing
    └── index.md

index.md looks like this:

Hello world!

writing/index.md is:

---
tags: writing
---
Writing

search/index.md contains:

---
tags: writing
---

If I run eleventy the search/solr.json looks like:

[{"url":"/writing/","date":"2019-05-08T20:55:15.146Z","content":"<p>Writing</p>\n"},{"url":"/search/","date":"2019-05-08T20:56:52.673Z","content":""}]

[Ryuno-Ki]

Looking into eleventys source code, I can find a few points, but not enough to tell what's going on just from reading …

The error you observe is only thrown here:

eleventy/src/TemplateMap.js

Line 367 in 192d6bf

throw new UsingCircularTemplateContentReferenceError(

That means three things:

1. a certain page must be part of usedTemplateContentTooEarlyMap`
2. getTemplateMapContent() must throw an error for this certain pageEntry
3. EleventyErrorUtil.isPrematureTemplateContentError(e) must be true`

If I follow 1, I can see that at some point it must have been part of orderedMap paramter, which in turn is part of orderedPaths. Here I stopped digging deeper.

2. getTemplateMapContent is defined in

   eleventy/src/Template.js

   Lines 654 to 663 in f452534

   	async getTemplateMapContent(pageMapEntry) {
   	pageMapEntry.template.setWrapWithLayouts(false);
   	let content = await pageMapEntry.template._getContent(
   	pageMapEntry.outputPath,
   	pageMapEntry.data
   	);
   	pageMapEntry.template.setWrapWithLayouts(true);
   	return content;
   	}

   
   Here I can find a reference on data, which might give a hint on what's going on (theory: LunrIndex is part of a collection itself and tries to parse itself).

3. isPrematureTemplateContentError is a small utility:
   

   eleventy/src/EleventyErrorUtil.js

   Lines 4 to 14 in f2de248

   	static isPrematureTemplateContentError(e) {
   	// TODO the rest of the template engines
   	return (
   	e instanceof TemplateContentPrematureUseError ||
   	(e.originalError &&
   	e.originalError.name === "RenderError" &&
   	e.originalError.originalError instanceof
   	TemplateContentPrematureUseError) || // Liquid
   	e.message.indexOf("TemplateContentPrematureUseError") > -1
   	); // Nunjucks
   	}

Have you run eleventy with DEBUG=Eleventy* eleventy to see, whether it contains anything meaningful?

[gerwitz]

This feels like a red herring, but it's the only lead I have until I create a test-specific project: seemingly all of my pages are being visited twice by at least one process:

  Eleventy:Template Rendering permalink for './src/site/search/lunr.json.11ty.js': /search/lunr.json becomes '/search/lunr.json' +10ms
  Eleventy:Template Rendering permalink for './src/site/search/lunr.json.11ty.js': /search/lunr.json becomes '/search/lunr.json' +1ms

[gerwitz]

FWIW, the current version of the template I'm struggling with can be found here.

I will attempt to replicate your working config and then break it ¯\_(ツ)_/¯

[jevets]

I haven't tried to access collections from within .11ty.js template files yet, but in the docs, there's this chunk, which tells me the template file itself will receive the collections object. data inside the class seems like it'd only be the local data.

module.exports = function({collections}) {
  return `<ul>
${collections.post.map((post) => `<li>${ post.data.title }</li>`).join("\n")}
</ul>`;
};

From https://www.11ty.io/docs/collections/

---

Have you tried something like this?

var lunr = require('lunr');

module.exports = ({ collections }) => {
  return class LunrIndex {
    data() {...}
    render() {...}
  }
}

See also this page in the docs, specifically the code block. I wonder if collections are available to this inside of data().

class Test {
  data() {
    return {
      title: "This is my blog post title",
      // Writes to "/this-is-my-blog-post-title/index.html"
      permalink: function(data) {
        return `/${this.slug(data.title)}/`;
      }
    };
  }

  render(data) { /* … */ }
}

module.exports = Test;

[jevets]

@gerwitz

(I'm playing with this some more to get familiar with using collections in 11ty.js templates.)

I'm not getting any circular template reference errors, whether eleventyExcludeFromCollections is true or false.

I'm wondering if your error stems from another spot in the render method.

My basic test class:

module.exports = class {
  data() {
    return {
      title: 'Test Page',
      eleventyExcludeFromCollections: true,
      // eleventyExcludeFromCollections: false,
    }
  }

  render(data) {
    let str = `<h1>${data.title}</h2>`
    data.collections.all.forEach(item => {
      str += `<p>${item.data.title} @ ${item.url}</p>`
    })
    return str
  }
}

It shows its own title and link when eleventyExcludeFromCollections=false and hides itself when eleventyExcludeFromCollections=true, as expected.

• Have you tried using data.collections.all directly in your call to lunr(function () {...}) and when building your docMap, as opposed to the pages variable?

[jevets]

@gerwitz

Also, I don't see the methods for this.ref('ref') and this.field('title') in the class definition. I'm not familiar with lunr, but I'd assume this is trying to reference methods on your instance of LunrIndex, and those methods aren't defined in your class:
https://github.com/gerwitz/hgc-v12/blob/master/src/site/search/lunr.json.11ty.js

Glancing at lunr, I'm guessing you may need to generate a JSON index from collections.all that contains page titles, text fields, etc., then use lunr on the client side to consume that JSON index, and that's where you'd specify ref and fields for lunr. But again I'm not familiar with lunr, so I may be off on this.

[gerwitz]

Thanks for helping me think through this!

The code in my repo is currently working (avoiding .all). So the this references to LunrIndex are fine, and the overall template works. Setting eleventyExcludeFromCollections=true does in fact hide it from collections.all yet still throws this error, which is what confuses me so much!

You’ve given me some good food for thought with where I perform the reference to .all, so I’ll be back with comments soon…

[jevets]

Sure, hope I can help at all.
It is that same repo, right? (specific branch or anything?) https://github.com/gerwitz/hgc-v12
I'll clone it and try it locally, see if I can find anything else to help.

[jevets]

I've followed you down the rabbit hole to find isPrematureTemplateContentError and UsingCircularTemplateContentReferenceError, as well.

I am able to reproduce the error you're getting, with a local clone of your repo.

I have, however, isolated the line that throws the error:

lunr(function() {
  this.add({
    ...
    // This throws the circular error,
    // even though the search page itself isn't being iterated over. 
    // The exclude from collections setting seems to be working, 
    // yet we're still getting the error.

    content: item.templateContent,
  })
}

Throwing a try/catch around that piece, something like this (setting a separate var for clarity):

pages.forEach(function (item, index) {
  try {
    const theTemplateContent = item.templateContent
  } catch (e) {
    console.log('caught error', e)
  }
})

Results in very many error log messages in the console. 752, actually, which is the same number of items in your lunr.json's map object. (I suspect that it's the same number of final output files.)

...
// lots more of the same stuff
caught error TemplateContentPrematureUseError: Tried to use templateContent too early (./src/site/library/princples/complexity.md)
caught error TemplateContentPrematureUseError: Tried to use templateContent too early (./src/site/library/princples/determinism.md)
// lots more of the same stuff
...

I'd start to suspect the source of these errors is coming from something at another level. (Not sure if any of this helps, hopefully it'll trigger some other idea to debug)

BTW you can try DEBUG=*Error* npx eleventy [--serve] to trim down debug output. Or DEBUG=*Error* yarn start.

[jevets]

For what it's worth (may help debug further):

I tried my 'Test Page' .11ty.js template file class from above to see if accessing item.templateContent in its render() method would throw the error.

• It does not throw the error when eleventyExcludeFromCollections is true
• It does throw the error when eleventyExcludeFromCollections is false (or unset)

Same UsingCircularTemplateContentReferenceError exception.

This seems like expected behavior since the templateContent is dependent on the very method that renders it, but in your case you're getting this error regardless of the setting, where in my case I'm avoiding the error since the page is excluded from collections.

module.exports = class {
  data() {
    return {
      title: 'Test Page',
      eleventyExcludeFromCollections: true, // no error
      // eleventyExcludeFromCollections: false, // throws the error
    }
  }

  render(data) {
    let str = ``
    data.collections.all.forEach(item => {
      str += `
        <pre>${item.inputPath}</pre>
        <textarea rows="10" cols="80">${item.templateContent}</textarea>
        <br><br>
      `
    })
    return str
  }
}

[gerwitz]

Welcome to the rabbit hole, and thanks for the DEBUG tips. I am simplifying my use of the Lunr and will look into it further.

Meanwhile, I simplified my scenario to a similar test, and still had this issue! So now I've copied your test template above and:

> ./src/site/test-jevets.11ty.js contains a circular reference (using collections) to its own templateContent. (UsingCircularTemplateContentReferenceError):
    UsingCircularTemplateContentReferenceError: ./src/site/test-jevets.11ty.js contains a circular reference (using collections) to its own templateContent.
        at TemplateMap.populateContentDataInMap (/Users/hans/Working/hgc-v12/node_modules/@11ty/eleventy/src/TemplateMap.js:367:17)
        at <anonymous>
        at process._tickCallback (internal/process/next_tick.js:118:7)

That's using your template above verbatim. (It is of course excluded from collections, and removing the call to .templateContent also prevents the exception.)

This is "spooky" and makes me suspect the TemplateContentPrematureUseError you noted is relevant.

[gerwitz]

Closer to root cause: excluding my "root" template (index.njk) from the collection prevents this error, as does removing its reference to .templateContent on another template.

This appears to be a case of the circular reference test being fooled, unless I'm the foolish one and am not seeing how /test.njk → /index.njk → /notes/2019/2019-05-12-this-story-about-ice-xviii-is.md is circular. 🤔

[gerwitz]

It looks like all those TemplateContentPrematureUseError catches are really just cache misses for a map of pre-layout template output.

@zachleat it looks to me like that cache introduced with f2de248 will try again for nested calls to .templateContent, but will only allow one pass. So by nesting more than one layer I broke it?

[zachleat]

Sorry—catching up here, y’all. I think you found a legitimate bug here. When Eleventy creates it’s content dependency graph (in TemplateMap: getMappedDependencies, getDelayedMappedDependencies, getPaginatedOverCollectionsMappedDependencies, and getPaginatedOverAllCollectionMappedDependencies methods) we don’t take into account eleventyExcludeFromCollections but we should!

It’s only taken into account when the collections are created (not in the graph to determine render order)

eleventy/src/TemplateMap.js

Line 254 in e127ca1

if (!map.data.eleventyExcludeFromCollections) {

[gerwitz]

Well, that’s a lot easier than I thought it’d be!

[zachleat]

@gerwitz what’s your NPM familiarity? Would you be comfortable pointing your package.json to a branch on this repo to test a fix?

[gerwitz]

My familiarity is just enough for that … give me a branch and I’ll test it out.

(This may take me a little time since I worked around this issue and the diagnostic tests are presently out of my reach.)

[zachleat]

Alternatively, can you make a branch on your own repo that shows the error? (or maybe point me to a commit that does it?)

[zachleat]

Which ever is easiest, here’s an attempt at a fix here: https://github.com/11ty/eleventy/tree/522

[zachleat]

If it doesn’t work, a broken branch in your repo would be much appreciated!

[gerwitz]

It works, thanks!

[zachleat]

I was very confused when this was closed! Thanks for that PR though :D