What other kinds of tests can the SILE project have?

[dontlaugh]

Hypothetically, are there more kinds of tests we can contribute?

Existing Tests

The existing files under tests/ take a "golden file" approach. This works great. I see there is an "update-actuals" make recipe as well, so diffs can be checked in, if they're valid. At least, that's what it looks like from glancing at the automake script 🤔

There are also Lua tests, which can - presumably - verify that you've constructed an expected object structure. Unit test stuff. Essential.

Possible Additional Tests

What else is possible? Given disk space and compute, could you use something like ImageMagick to convert rendered PDFs into images and then do diffs on actual image "golden files"?

Are there other kinds of structural tests that can bridge the gap (or test some middle ground) between pure Lua and rendered PDF?

[alerque]

At the moment we don't automatically test rendered PDFs at all. We have a dummy backend that outputs debug information about what the PDF backend is being asked to render in plain text format with some relevant details. That's what we diff as part of our end to end regression tests.

Usually the debug information is far more informative than visually comparing PDFs because it gives a strong clue about what exactly changed. Sometimes however a visual is good. I've been using diff-pdf to compare before and after PDFs when I need to check a regression manually. There are some make targets to facilitate this, but I also just use some of my workflow tooling locally. I even have some over/montage variants that take PDFs from all the tests and render them together as proof sheets (again from book publishing tooling I have locally).

As noted we're using Busted for unit tests of smaller chunks of the Lua API. Honestly we need more of these and fewer full regression tests as they are more informative and easier to maintain. Quite a few things we have as regression tests could probably be reduced to unit tests.

We have three holes in the testing I can think of:

• The CLI is not being tested beyond what the regression test script uses (basically no options). It would be useful to test that CLI options are parsed and used correctly ... not so much because the CLI breaks easily but because we're not really excersizing the code paths that can be triggered that way. For example --use inputter.xyz can be used to load an inputter module before parsing a document. That whole interface of modules being loaded before documents isn't being testing yet.

• The actual PDF output isn't being tested at all. I don't want to store PDF golden masters in the code repository except perhaps on a carefully selected basis, but we could at least generate them and run some checks on the result. We already require pdfinfo as part of the build for the make selfcheck target. A large number of the tests with no expectations yet (random example: tests/rotate.sil) don't have expectations because they don't work in the debug backend. Either the debug backend needs to be improved or we need to adapt the regression tester to check at least for valid PDF output (even if we don't same the visuals for comparison).

• The third major hole now is that 3rd party modules don't have an easy to way test themselves. I would like to do something to make 3prd party LuaRocks packaged modules easy to test against SILE, and possibly use SILE's CI to trigger some of them to test that we don't break downstream things. This may mean writing a busted helper, or having a script downstream projects can use to run our regression style tests or something.

[Omikhleia]

I would like to do something to make 3prd party LuaRocks packaged modules easy to test against SILE, and possibly use SILE's CI to trigger some of them to test that we don't break downstream things

Convert to a dedicated issue and close this discussion?
(What do we expect further from it?)

Maybe: "The actual PDF output isn't being tested at all."
We could also have an HTML outputter and diff on that. Should we?

[dontlaugh]

Since I opened this, I can say that I am okay with converting this discussion to an issue.

I am not using SILE right now, but as I recall adding some end to end tests would be good. It's a hard problem, but the upside could be huge.

[Omikhleia]

Doing some triage, it seems that #949 and #391 might be on a similar topic.