Fix markdown math brackets render problem

[charles7668]

close #31371

compare

original text:

$i = \sqrt{-1}$ ($i = \sqrt{-1}$)

$i = \sqrt{-1}$ [$i = \sqrt{-1}$]

$i = \sqrt{-1}$ {$i = \sqrt{-1}$}

$abcd

(this is a test)

[$test]

$test$

$a$ ($b$) [$c$] {$d$}

before fix:

after fix:

[wxiaoguang]

Thank you for the fix!

ps: I will edit the PR title to remove the (# xxx) part: that part is only used for backporting in Gitea.