Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Hakyll.Web.Pandoc.Biblio.Tests failing #1002

Closed
Vekhir opened this issue Sep 12, 2023 · 8 comments · Fixed by #1003
Closed

Hakyll.Web.Pandoc.Biblio.Tests failing #1002

Vekhir opened this issue Sep 12, 2023 · 8 comments · Fixed by #1003
Labels
bug

Comments

@Vekhir
Copy link

Vekhir commented Sep 12, 2023

Hi!
When trying to build hakyll 4.16.1.0, I get the following test failures:

  Hakyll.Web.Pandoc.Biblio.Tests
    biblio01:              FAIL (0.01s)
      Test output was different from 'tests/data/biblio/cites-meijer-pandoc3.golden'. It was:
      <!doctype html>
      <html lang="en">
          <head>
              <meta charset="utf-8">
              <title>This page cites a paper.</title>
          </head>
          <body>
              <h1>This page cites a paper.</h1>
              <p>I would like to cite one of my favourite papers <span class="citation" data-cites="meijer1991functional">(Meijer, Fokkinga, and Paterson 1991)</span> here.</p>
      <div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0" role="list">
      <div id="ref-meijer1991functional" class="csl-entry" role="listitem">
      Meijer, Erik, Maarten Fokkinga, and Ross Paterson. 1991. <span>“Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire.”</span> In <em>Conference on Functional Program>
      </div>
      </div>
          </body>
      </html>

      Use -p '/biblio01/' to rerun this test only.
    biblio02:              FAIL
      Test output was different from 'tests/data/biblio/cites-meijer-pandoc3.golden'. It was:
      <!doctype html>
      <html lang="en">
          <head>
              <meta charset="utf-8">
              <title>This page cites a paper.</title>
          </head>
          <body>
              <h1>This page cites a paper.</h1>
              <p>I would like to cite one of my favourite papers <span class="citation" data-cites="meijer1991functional">(Meijer, Fokkinga, and Paterson 1991)</span> here.</p>
      <div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0" role="list">
      <div id="ref-meijer1991functional" class="csl-entry" role="listitem">
      Meijer, Erik, Maarten Fokkinga, and Ross Paterson. 1991. <span>“Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire.”</span> In <em>Conference on Functional Program>
      </div>
      </div>
          </body>
      </html>

      Use -p '/biblio02/' to rerun this test only.
    biblio03:              FAIL
      Test output was different from 'tests/data/biblio/cites-multiple-pandoc3.golden'. It was:
      <!doctype html>
      <html lang="en">
          <head>
              <meta charset="utf-8">
              <title>This page cites a paper and a book.</title>
          </head>
          <body>
              <h1>This page cites a paper and a book.</h1>
              <p>I would like to cite one of my favourite papers <span class="citation" data-cites="meijer1991functional">(Meijer, Fokkinga, and Paterson 1991)</span> here.</p>
      <p>And also a book <span class="citation" data-cites="lipovaca2012">(Lipovača 2012)</span>.</p>
      <div id="refs" class="references csl-bib-body hanging-indent" data-entry-spacing="0" role="list">
      <div id="ref-lipovaca2012" class="csl-entry" role="listitem">
      Lipovača, Miran. 2012. <em>Learn You a Haskell for Great Good! A Beginner’s Guide</em>. San Francisco, CA: No Starch Press.
      </div>
      <div id="ref-meijer1991functional" class="csl-entry" role="listitem">
      Meijer, Erik, Maarten Fokkinga, and Ross Paterson. 1991. <span>“Functional Programming with Bananas, Lenses, Envelopes and Barbed Wire.”</span> In <em>Con<truncated>
      Use --accept or increase --size-cutoff to see full output.
      Use -p '/biblio03/' to rerun this test only.

All other ones are fine. I'm running pandoc 3.1.8

System information

OS: Arch Linux
Kernel: Linux 6.4.12-arch1-1
GHC: 9.0.2
haskell-hakyll: 4.16.1.0
haskell-pandoc: 3.1.8
@Minoru
Copy link
Collaborator

Minoru commented Sep 12, 2023

Also on Stackage: commercialhaskell/stackage#7110

I don't have time to look into this right now, but if anyone else does: see if Pandoc's bibtex code changed recently. Our tests validate that the output didn't change, and it failed previously when Pandoc 3 was released. Mabe it's the same thing again.

@Minoru Minoru added the bug label Sep 12, 2023
@Vekhir
Copy link
Author

Vekhir commented Sep 14, 2023

The tests work with pandoc 3.1.7, so it's at least a very recent change

@Minoru
Copy link
Collaborator

Minoru commented Sep 14, 2023

Thanks for the pointer! Looking at the full diff in the tests, I notice that a new HTML attribute, data-entry-spacing, was added to some <div>s. The changelog for 3.1.8 indeed says:

Text.Pandoc.Citeproc: always include an entry-spacing attribute in the Div if the bibliography element contains an entry-spacing attribute (previously we omitted it when it was 0) (jgm/pandoc#9058).

The easiest path forward is to update our tests and put a lower boundary on Pandoc so the updated test can only be built with Pandoc > 3.1.7. However, this would make Hakyll unbuildable on older systems that haven't updated Pandoc yet, which seems excessively bad given it's only because of some niche attribute.

Another way is to use different golden files for different Pandoc versions. We already do that for 2.x.y vs 3.x.y, we could add a third option. Not looking forward to doing that every time something in Pandoc changes though.

Any more ideas?

@Vekhir
Copy link
Author

Vekhir commented Sep 15, 2023

The third option would be to not allow pandoc 3.1.8 or later, but that's also not viable longterm.

Ultimately, the issue here is that the tests are too brittle - they test too strictly that the output is what we want. But for visual stuff, that's the only option I guess.
I've briefly thought about a programmatic way of checking for "equivalence" (more general than string comparison, like removing data-entry-spacing or something), but ultimately, every change in pandoc would probably require a change here too.

So for now, a different golden file is probably the way to go. This time it's only one test, so maybe it won't be too difficult to maintain?

@jaspervdj
Copy link
Owner

jaspervdj commented Sep 15, 2023 •
edited
Loading

I agree that constraining pandoc (up or down) is not the way to go.

The viable options are:

  1. Adding some post-process functions that we run before comparing (and then add a single one to strip out this tag, since we don't really care about it in the tests)
  2. Add specific tests for every pandoc version... if we do that, we need to make it very easy to add new versions to make maintenance easier
  3. Rather than golden tests; check that specific substrings are in the output (e.g. the actual reference)

I have a slight preference towards (2) but only if we can make it easy enough to do this. Happy to work on this but interested to hear if @Minoru leans more towards (1) or (2).

@Minoru
Copy link
Collaborator

Minoru commented Sep 18, 2023

I prefer (2), and it's easy enough to implement with the preprocessor; I'll have a PR ready in a bit.

Minoru added a commit to Minoru/hakyll that referenced this issue Sep 18, 2023
@Minoru
Fix Biblio tests with Pandoc 3.1.8 (jaspervdj#1002)
70e414a
Minoru added a commit to Minoru/hakyll that referenced this issue Sep 18, 2023
@Minoru
Fix Biblio tests with Pandoc 3.1.8
d12e1d4 
Fixes jaspervdj#1002.
@Minoru Minoru mentioned this issue Sep 18, 2023
Merged
Minoru added a commit that referenced this issue Sep 20, 2023
@Minoru
Fix Biblio tests with Pandoc 3.1.8 (#1003)
3b48802 
* Fix Biblio tests with Pandoc 3.1.8

Fixes #1002.

* Update hakyll.cabal with new golden file names
@Minoru
Copy link
Collaborator

Minoru commented Sep 20, 2023 •
edited
Loading

The fix is now merged and released as part of 4.16.2.0. I briefly considered making a revision to 4.16.1.0 to forbid building against newer Pandocs, but decided against it because the incompatibility only affects the tests.

@Vekhir
Copy link
Author

Vekhir commented Sep 20, 2023

Thank you very much! I think you meant 4.16.2.0, the new release is much appreciated!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix Biblio tests with Pandoc 3.1.8 Minoru/hakyll
3 participants
@jaspervdj @Minoru @Vekhir