t/s.t fails and crashes with Perl 5.32.0 randomly

[ppisar]

After upgrading Perl from 5.30.3 to 5.32.0 I observe random failures in t/s.t test:

t/s.t ..................... 1/10 
#   Failed test 's/.//g; 'abc' => '''
#   at t/s.t line 14.
#          got: 'c'
#     expected: ''

#   Failed test 's/./1+1/eg; 'abc' => '222''
#   at t/s.t line 34.
#          got: '22c'
#     expected: '222'

#   Failed test 's/(.)/$1+1/eg; '123' => '234''
#   at t/s.t line 42.
#          got: '233'
#     expected: '234'
# Looks like you failed 3 tests of 10.
t/s.t ..................... Dubious, test returned 3 (wstat 768, 0x300)
Failed 3/10 subtests 

or crashes:

1..10
ok 1 - s/a//; 'ab' => 'b'
ok 2 - s/.//g; 'abc' => ''
ok 3 - s/.//g; 'abcd' => ''
ok 4 - s/.//g; 'abcdefg' => ''
ok 5 - s/..//g; 'aabbc' => 'c'
ok 6 - s/./1+1/eg; 'a' => '2'
ok 7 - s/./1+1/eg; 'abc' => '222'
ok 8 - s/(.)/$1+1/eg; '3' => '4'
ok 9 - s/(.)/$1+1/eg; '123' => '234'
ok 10 - s/([0-9]+)/$1*2/e; 'abc123xyz' => ''abc246xyz'
free(): invalid size
Aborted (core dumped)

or:

1..10
ok 1 - s/a//; 'ab' => 'b'
ok 2 - s/.//g; 'abc' => ''
ok 3 - s/.//g; 'abcd' => ''
ok 4 - s/.//g; 'abcdefg' => ''
ok 5 - s/..//g; 'aabbc' => 'c'
ok 6 - s/./1+1/eg; 'a' => '2'
ok 7 - s/./1+1/eg; 'abc' => '222'
ok 8 - s/(.)/$1+1/eg; '3' => '4'
ok 9 - s/(.)/$1+1/eg; '123' => '234'
ok 10 - s/([0-9]+)/$1*2/e; 'abc123xyz' => ''abc246xyz'
free(): invalid pointer
Aborted (core dumped)

This smells like a failure in a memory management.

[ppisar]

I can reduce it to:

$ cat t/s.t 
use strict;
use Test::More tests => 2;
use re::engine::PCRE2;

$_ = "ab";
$_ =~ s/a//;
is($_, 'b', q(s/a//; 'ab' => 'b'));

$_ = "a";
$_ =~ s/./2/e;
is($_, "2", q(s/./1+1/eg; 'a' => '2'));

[test@fedora-33 re-engine-PCRE2-0.16]$ while (perl -Mblib t/s.t); do :; done
1..2
ok 1 - s/a//; 'ab' => 'b'
ok 2 - s/./1+1/eg; 'a' => '2'
1..2
ok 1 - s/a//; 'ab' => 'b'
ok 2 - s/./1+1/eg; 'a' => '2'
1..2
ok 1 - s/a//; 'ab' => 'b'
ok 2 - s/./1+1/eg; 'a' => '2'
free(): double free detected in tcache 2
Aborted (core dumped)

It's triggered by s///e substituting on a variable that was substituted before. It does not have to be the implicit $_. It can be any other variable. Removing the /e flag disappears the bug.

[ppisar]

Valgrind reports on each run:

$ while (valgrind -- perl -Mblib t/s.t); do :; done
[...]
1..2
ok 1 - s/a//; 'ab' => 'b'
ok 2 - s/./1+1/eg; 'a' => '2'
==29384== Invalid free() / delete / delete[] / realloc()
==29384==    at 0x483A9F5: free (vg_replace_malloc.c:538)
==29384==    by 0x4913A5A: Perl_pregfree2 (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x4975859: Perl_sv_clear (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x4976001: Perl_sv_free2 (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x48AED22: Perl_op_clear (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x48AEE0C: Perl_op_free (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x48D138D: perl_destruct (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x109323: main (perlmain.c:138)
==29384==  Address 0x6785510 is 0 bytes inside a block of size 10 free'd
==29384==    at 0x483A9F5: free (vg_replace_malloc.c:538)
==29384==    by 0x4970003: Perl_pp_subst (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x4967875: Perl_runops_standard (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x48D6C83: perl_run (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x109349: main (perlmain.c:127)
==29384==  Block was alloc'd at
==29384==    at 0x4839809: malloc (vg_replace_malloc.c:307)
==29384==    by 0x49477BC: Perl_safesysmalloc (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x497F9A7: Perl_sv_grow (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x4986B07: Perl_sv_setpvn (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x4986CD7: Perl_newSVpvn_flags (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x496F947: Perl_pp_subst (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x4967875: Perl_runops_standard (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x48D6C83: perl_run (in /usr/lib64/libperl.so.5.32.0)
==29384==    by 0x109349: main (perlmain.c:127)

[ppisar]

Here a the line numbers:

==81403== Invalid free() / delete / delete[] / realloc()
==81403==    at 0x483A9F5: free (vg_replace_malloc.c:538)
==81403==    by 0x4906A5A: Perl_pregfree2 (regcomp.c:21623)
==81403==    by 0x4968859: Perl_sv_clear (sv.c:6670)
==81403==    by 0x4969001: Perl_sv_free2 (sv.c:7132)
==81403==    by 0x48A1D22: Perl_SvREFCNT_dec (inline.h:235)
==81403==    by 0x48A1D22: Perl_op_clear (op.c:1151)
==81403==    by 0x48A1E0C: Perl_op_free (op.c:957)
==81403==    by 0x48A1E0C: Perl_op_free (op.c:847)
==81403==    by 0x48C438D: perl_destruct (perl.c:881)
==81403==    by 0x109323: main (perlmain.c:138)
==81403==  Address 0x6530da0 is 0 bytes inside a block of size 10 free'd
==81403==    at 0x483A9F5: free (vg_replace_malloc.c:538)
==81403==    by 0x4963003: Perl_pp_subst (pp_hot.c:4501)
==81403==    by 0x495A875: Perl_runops_standard (run.c:41)
==81403==    by 0x48C9C83: S_run_body (perl.c:2761)
==81403==    by 0x48C9C83: perl_run (perl.c:2684)
==81403==    by 0x109349: main (perlmain.c:127)
==81403==  Block was alloc'd at
==81403==    at 0x4839809: malloc (vg_replace_malloc.c:307)
==81403==    by 0x493A7BC: Perl_safesysmalloc (util.c:155)
==81403==    by 0x49729A7: Perl_sv_grow (sv.c:1616)
==81403==    by 0x4979B07: Perl_sv_setpvn (sv.c:4993)
==81403==    by 0x4979CD7: Perl_newSVpvn_flags (sv.c:9352)
==81403==    by 0x4962947: Perl_pp_subst (pp_hot.c:4433)
==81403==    by 0x495A875: Perl_runops_standard (run.c:41)
==81403==    by 0x48C9C83: S_run_body (perl.c:2761)
==81403==    by 0x48C9C83: perl_run (perl.c:2684)
==81403==    by 0x109349: main (perlmain.c:127)

[ppisar]

The failure happens since this perl commit Perl/perl5@f0ac8a3:

commit f0ac8a377ab01ad6936776b0c73d0c1eab98f782 (refs/bisect/bad)
Author: Karl Williamson <[email protected]>
Date:   Wed Feb 12 16:40:29 2020 -0700

    op.h: Move some flag bits down
    
    This is in preparation for adding a new flag bit at the end in a future
    commit.  It could have been added in the unused space that the first of
    these was moved to, but the new one is less important/used, so I thought
    it best to come last.  The reason to use unused space is to preserve
    binary compatibility with the bits, and we don't care about that at this
    point in the development cycle.

[rurban]

Introduced with v5.31.9

gdb bt

$ gdb --args perl5.32.0d-nt -Iblib/arch -Iblib/lib t/s.t

double free or corruption (out)

Program received signal SIGABRT, Aborted.
0x00007ffff7c4c9e5 in raise () from /lib64/libc.so.6
(gdb) bt
#0  0x00007ffff7c4c9e5 in raise () from /lib64/libc.so.6
#1  0x00007ffff7c35895 in abort () from /lib64/libc.so.6
#2  0x00007ffff7c90857 in __libc_message () from /lib64/libc.so.6
#3  0x00007ffff7c97d7c in malloc_printerr () from /lib64/libc.so.6
#4  0x00007ffff7c993c0 in _int_free () from /lib64/libc.so.6
#5  0x0000000000597839 in Perl_safesysfree (where=0x1149310) at util.c:399
#6  0x0000000000573652 in Perl_pregfree2 (rx=0x112e640) at regcomp.c:21623
#7  0x000000000063be8f in Perl_sv_clear (orig_sv=0x112e640) at sv.c:6670
#8  0x000000000063effd in Perl_sv_free2 (sv=0x112e640, rc=1) at sv.c:7132
#9  0x000000000042264e in Perl_SvREFCNT_dec (sv=0x112e640) at inline.h:235 (REFCNT=0)
#10 0x000000000042608b in Perl_op_clear (o=0x1129a98) at op.c:1157 (pp_subst)
#11 0x0000000000425bb1 in Perl_op_free (o=0x1129a98) at op.c:957
#12 0x00000000004673f4 in perl_destruct (my_perl=0xa802a0) at perl.c:881
#13 0x0000000000420179 in main (argc=4, argv=0x7fffffffd268, env=0x7fffffffd290) at perlmain.c:138

I dont see yet any fault in f0ac8a377ab01ad6936776b0c73d0c1eab98f782, more likely in the next commit.

[demerphq]

This is fixed by #40

The problem is the code was pointer copying the strbeg during exec, which then causes problems. Using savepvn() solves the problem at some cost. Really you should be copying the COW logic.