Plain text writer does not preserve line breaks

[jeffvalk]

The plain text writer coalesces sequential LineBreaks into a single new line.

The following example:

json='{"pandoc-api-version":[1,23,1],"meta":{},"blocks":[{"t":"Plain","c":[{"t":"Str","c":"two_line_breaks:"},{"t":"LineBreak"},{"t":"LineBreak"},{"t":"Str","c":"four_line_breaks:"},{"t":"LineBreak"},{"t":"LineBreak"},{"t":"LineBreak"},{"t":"LineBreak"},{"t":"Str","c":"last_line"}]}]}'
echo $json | pandoc -f json -t plain

produces this output:

two_line_breaks:
four_line_breaks:
last_line

where I would expect this output:

two_line_breaks:

four_line_breaks:



last_line

This is similar to the CSV reader issue (#9797) fixed in fa01764.

[jeffvalk]

It may be worth noting that I'm using the plain text writer in a Lua filter as a whitespace-preserving version of the pandoc.utils.stringify function:

-- element contains raw markdown to parse
local s = pandoc.write(pandoc.Pandoc(element), "plain") -- write element AST to plain text
local b = pandoc.read(s, "markdown").blocks             -- read text as markdown

If there's a more idomatic way to do this, please let me know!

[jgm]

This is because blank lines mean "new paragraph."

You can always include a RawInline or RawBlock element with whatever you want, and it will pass directly through to the output format as long as the format of the RawINline/Block matches...

[jgm]

White space won't be preserved anyway; e.g. consecutive spaces would be collapsed.

[avidseeker]

I had a similar request: #9663

It would be nice if Pandoc was less aggressive in reformating markdown.

You should not expect the markdown writer to preserve aspects of your document that don't relate to its abstract structure.

[jeffvalk]

This is because blank lines mean "new paragraph."

I'd fully expect this whenever a writer's format specifies it (e.g. markdown), but is this true for "plain" text? I presumed the plain text writer would output text without applying any format's semantics or production rules. I tend to think this should be the case; and if it intentionally isn't, the semantics attached to "plain" text should be documented.

White space won't be preserved anyway; e.g. consecutive spaces would be collapsed.

This depends on the reader and the specification it implements. The JSON snippet above was produced by Pandoc's CSV reader. As of fa01764, the CSV reader preserves multiple consecutive LineBreaks, which is the correct behavior according to RFC 4180 (see #9797).

[jgm]

Our plain output is essentially a variant of markdown. Of course, there is no particular thing that "plain" text means.

I presumed the plain text writer would output text without applying any format's semantics or production rules.

Block-level formatting has to happen somehow. How do we represent a paragraph in plain text? A heading? Lists?

Think of plain as a stripped-down markdown that avoids using symbols to mark syntax.

Multiple consecutive LineBreaks can be rendered in some formats, e.g. HTML, and even in pandoc's markdown, which has the BACKSLASH+NEWLINE way of writing a line break. But not in formats (like plain) where (a) a line break is just a newline and a blank line is a paragraph break.

[jgm]

As I hinted, though, there's an easy workaround with a Lua filter. Just have the filter replace LineBreak elements with RawInline (Format "plain") "\n".

[jeffvalk]

Our plain output is essentially a variant of markdown.

Thanks, @jgm. I think this would be good to document.

Think of plain as a stripped-down markdown that avoids using symbols to mark syntax.

This description of the format in the manual would clarify nicely.

[jeffvalk]

As I hinted, though, there's an easy workaround with a Lua filter. Just have the filter replace LineBreak elements with RawInline (Format "plain") "\n".

This should certainly work, and I appreciate the workaround. I do think a function for getting the text of the AST while preserving whitespace would be generally useful in Lua filters. I opened a separate issue (#10015) to propose that.

[jgm]

Preserving whitespace is not possible in a filter, because the whitespace information is often lost in the parsers.

[jeffvalk]

It's possible for a reader to encode whitespace in the AST; and depending on the language/format specification the reader implements, this may be required for correctness. #9797 was an instance of this in a built-in reader. And custom readers could certainly introduce others.