Regression: combined shorthand flags (like -so for -s -o) fail as of #8879

[hftf]

To reproduce:

$ pandoc -so tmp
value must be either true or false
$ pandoc --version
pandoc 3.1.5

Filing a separate bug as requested. See original topic at #8879 (comment).

[jgm]

As noted in #8879 I think we might have to live with this, but we can leave this open for comments.

[jgm]

A recent change will make the error message for the above command more intelligible:

    Argument of --standalone/-s must be either true or false

[hftf]

Speaking for myself, I have ten+ years of scripts/projects/Makefiles that invoke Pandoc and use combined short flags and am not looking forward to finding and updating them all :\

[jgm]

I'd really like to find a solution, but I don't know what the best solution is.
Here are some options:

1. Do nothing. Cons: regression from earlier CLI interface, may cause problems for existing scripts.
2. Roll back the change adding the optional boolean argument to all the boolean flags. Cons: This would be a breaking change the CLI interface, and there was a good motivation for that change.
3. Try to change the code so that short options can't take optional arguments. (This could be done by preprocessing the arguments before passing them to the option parser, perhaps, and splitting -so into -s and -o.) Cons: This would mean that you couldn't do -strue (probably fine as I doubt anyone does that) but also that you couldn't do e.g. -tmarkdown (which I'm sure people do).

[jgm]

Note: the following boolean flags have short versions:

-s/--standalone
-p/--preserve-tabs
-i/--incremental

That's only three. So perhaps there are a few more solutions;

4. Roll back the change just for these three options. Cons: awkward to have to remember that these are exceptions, and there are cases where we'd want to use the optional arguments for them too.
5. In the OptDescr list we pass to the option parser, provide separate clauses for s and standalone (and similarly for the other two). This would allow us to disallow the optional arguments for the short option names, while allowing them for the long ones. Cons: these would appear on separate lines in the --help output. Also, one would have to remember that one can't use the boolean options with short option names.
6. Variant of 3 above: preprocess arguments, but only do the splitting for short options that take optional boolean arguments or no arguments. Thus -tmarkdown would still work but not -strue. Cons: a bit messy to do this kind of preprocessing. At this point we might as well just do option parsing by hand, since in effect we'd be going through things twice!
7. Variant of 6, but allow -strue and -sfalse. Why not if we're preparsing the arguments anyway?
8. Write our own option parsing code, or find another library that does exactly what we want. Cons: many.

[jgm]

Sketch of solution 7:

Iterate through the raw list of arguments:

• If the argument is plain --, add it and all subsequent arguments to the new argument list
• If it begins with a single - (not followed by -): add the result of applying splitArg to the remainder of the argument to the new argument list
• Otherwise add it to the new argument list.

splitArg (c:cs)

• if c is a single-letter option that takes an optional boolean argument (i.e. s, p, or i, but we should construct this list from the option description list rather than hardcoding it), then:
  • if cs == "true" or "false" (case-insensitive), then return the list ['-':c:cs]
  • otherwise, return ['-',c] : splitArgs cs
• otherwise return ['-':c:cs]

[hftf]

I don't have much to contribute on the parsing front, but I could try to collect all the command-line invocations I have for use as parsing tests if needed. When I see major pull requests accepted without any tests whatsoever in a large project, it leaves me concerned.

[jgm]

Sure, that would be helpful. I'm trying to implement suggestion 7.

[jgm]

I've included some tests, but if you have cases that seem significantly different, please feel free to suggest them.

[hftf]

Sorry, can you point me toward the tests? I didn't find them in bec5429.

Sure, that would be helpful.

I did a cursory grep of my scripts, Makefiles, and terminal history and only see a few instances of combined short options. Stuff like pandoc -sN --number-offset=1 --toc --toc-depth=2 instructions.md > instructions.html is attested in the terminal history, and sometimes in Makefiles like pandoc -so $@ $< --toc --filter pandoc-citeproc. I consistently put a space before any option argument, it seems, so I never have anything like -frst -thtml but rather -f rst -t html. It wasn't an exhaustive search, though, since I didn't check on other machines and a lot of terminal history wasn't captured.

[jgm]

Sorry, forgot to check in the tests.