nested shortcode with codeblock rendering issue

[davidejones]

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.71.1-A301F6B2 darwin/amd64 BuildDate: 2020-05-25T09:09:07Z

Does this issue reproduce with the latest release?

Yes

What is the issue?

Nesting a markdown shortcode within a markdown shortcode can in some cases cause a codefence to render some of its contents as markdown.

This doesn't appear to happen if there is just one shortcode surrounding the markdown. e.g removing {{< first >}} and {{< /first >}} from the test example below.

Appears to happen with both methods of specifying markdown:

• {{% ... %}}
  OR
• {{< ... >}} with .Inner | markdownify

Screenshot of issue

Test

Bit of a messy test but you get the point
I added this to hugolib/shortcode_test.go and the line saying # This declares a view into HAProxy statistics, on port 3833 gets rendered as a h1 even though its inside a code block.

func TestNestedShortCodeWithCodeFence(t *testing.T) {
	t.Parallel()
	c := qt.New(t)

	builder := newTestSitesBuilder(t)
	builder.WithConfigFile("yaml",
		`
pygmentsuseclasses: true
pygmentsCodeFences: true

markup:
  goldmark:
    renderer:
      unsafe: true
  highlight:
    guessSyntax: true
`)

	builder.WithContent("page.md", `---
title: "Hugo Rocks!"
---

{{< first >}}

# This is first markdown

{{< second >}}

# This is second markdown

`+"```"+`
# Basic configuration
timeout client 5s
timeout server 5s
timeout connect 5s

# This declares a view into HAProxy statistics, on port 3833
mode http
stats enable
`+"```"+`
{{< /second >}}
{{< /first >}}
`)
	builder.WithTemplatesAdded("layouts/shortcodes/first.html", `<div>{{- .Inner | markdownify -}}</div>`)
	builder.WithTemplatesAdded("layouts/shortcodes/second.html", `<div>{{ .Inner | markdownify }}</div>`)
	builder.Build(BuildCfg{})

	s := builder.H.Sites[0]
	c.Assert(len(s.RegularPages()), qt.Equals, 1)

	builder.AssertFileContent("public/page/index.html", `<div><h1 id="this-is-first-markdown">This is first markdown</h1>
<div><h1 id="this-is-second-markdown">This is second markdown</h1>
<div class="highlight"><pre class="chroma"><code class="language-fallback" data-lang="fallback"># Basic configuration
timeout client 5s
timeout server 5s
timeout connect 5s

# This declares a view into HAProxy statistics, on port 3833
mode http
stats enable
</code></pre></div></div></p>
</div>`)
}

[moorereason]

Confirmed. The test content can be reduced to:

---
title: "Hugo Rocks!"
---
{{< first >}}
# This is first markdown
{{< second >}}
# This is second markdown
```
# COMMENT 1

# COMMENT 2
```
{{< /second >}}
{{< /first >}}

The key is to have a blank line in the code section.

[davidejones]

Thank you, much more concise! and yes we actually removed the spacing as a workaround to stop the issue.

[satotake]

s1 := `{{< first >}}
# This is first markdown

{{< second >}}

# This is second markdown

`+"```"+`
# Comment 1

# Comment 2
`+"```"+`
{{< /second >}}
{{< /first >}}
`

After inner markdownify, s1 is transformed like this as we expect

# This is first markdown

<div><h1 id="this-is-second-markdown">This is second markdown</h1>
<pre><code># Comment 1

# Comment 2
</code></pre></div>

According to commonmark, the above results in <h1> inside <pre>

Although this is a little surprise, it might not be a bug.

@davidejones
Is it fine for you to remove inner markdownify?
I mean

	builder.WithTemplatesAdded("layouts/shortcodes/first.html", `<div>{{- .Inner | markdownify -}}</div>`)
	builder.WithTemplatesAdded("layouts/shortcodes/second.html", `<div>{{ .Inner }}</div>`)

This is the easiest way to fix this, I think.