feat(gnovm): handle loop variables

[jaekwon]

Problem Definition

The problem originates from the issue described in #1135. While the full scope of the issue is broader, it fundamentally relates to the concept of loop variable escapes block where it's defined.

e.g. 1:

package main

import "fmt"

var s1 []*int

func forLoopRef() {
	defer func() {
		for i, e := range s1 {
			fmt.Printf("s1[%d] is: %d\n", i, *e)
		}
	}()

	for i := 0; i < 3; i++ {
		z := i + 1
		s1 = append(s1, &z)
	}
}

func main() {
	forLoopRef()
}

e.g. 2:

package main

type f func()

var fs []f

func forLoopClosure() {
	defer func() {
		for _, f := range fs {
			f()
		}
	}()

	for i := 0; i < 3; i++ {
		z := i
		fs = append(fs, func() { println(z) })
	}
}

func main() {
	forLoopClosure()
}

e.g. 3:

package main

func main() {
	c := 0
	closures := []func(){}
loop:
	i := c
	closures = append(closures, func() {
		println(i)
	})
	c += 1
	if c < 10 {
		goto loop
	}

	for _, cl := range closures {
		cl()
	}
}

Solution ideas

• identify escaped vars in preprocess:
  Detect situations where a loop variable is defined within a loop block(including for/range loops or loops constructed using goto statements), and escapes the block where it's defined.

• runtime allocation:
  Allocate a new heap item for the loop variable in each iteration to ensure each iteration operates with its unique variable instance.

• NOTE1: this is consistent with Go's Strategy:
  "Each iteration has its own separate declared variable (or variables) [Go 1.22]. The variable used by the first iteration is declared by the init statement. The variable used by each subsequent iteration is declared implicitly before executing the post statement and initialized to the value of the previous iteration's variable at that moment."

• NOTE2: the loopvar feature of Go 1.22 is not supported in this version, and will be supported in next version.

  not supporting capture i defined in for/range clause;

	for i := 0; i < 3; i++ {
		s1 = append(s1, &i)
	}

Implementation Details

Preprocess Stage(Multi-Phase Preprocessor):

• Phase 1: initStaticBlocks: Establish a cascading scope structure where predefine is conducted. In this phase Name expressions are initially marked as NameExprTypeDefine, which may later be upgraded to NameExprTypeHeapDefine if it is determined that they escape the loop block. This phase also supports other processes as a prerequisite#2077.

• Phase 2: preprocess1: This represents the original preprocessing phase(not going into details).

• Phase 3: findGotoLoopDefines: By traversing the AST, any name expression defined in a loop block (for/range, goto) with the attribute NameExprTypeDefine is promoted to NameExprTypeHeapDefine. This is used in later phase.

• Phase 4: findLoopUses1: Identify the usage of NameExprTypeHeapDefine name expressions. If a name expression is used in a function literal or is referrnced(e.g. &a), and it was previously defined as NameExprTypeHeapDefine, the used name expression is then given the attribute NameExprTypeHeapUse. This step finalizes whether a name expression will be allocated on the heap and used from heap.
  Closures represent a particular scenario in this context. Each closure, defined by a funcLitExpr that captures variables, is associated with a HeapCaptures list. This list consists of NameExprs, which are utilized at runtime to obtain the actual variable values for each iteration. Correspondingly, within the funcLitExpr block, a list of placeholder values are defined. These placeholders are populated during the doOpFuncLit phase and subsequently utilized in the doOpCall to ensure that each iteration uses the correct data.

• Phase 5: findLoopUses2: Convert non-loop uses of loop-defined names to NameExprTypeHeapUse. Also, demote NameExprTypeHeapDefine back to NameExprTypeDefine if no actual usage is found. Also , as the last phase, attributes no longer needed will be cleaned up after this phase.

Runtime Stage:

1. Variable Allocation:

   • Modify the runtime so that encountering a NameExprTypeHeapDefine triggers the allocation of a new heapItemValue for it, which will be used by any NameExprTypeHeapUse.

2. Function Literal Handling:

   • During the execution of doOpFuncLit, retrieve the HeapCapture values (previously allocated heap item values) and fill in the placeholder values within the funcLitExpr block.
   • When invoking the function (doOpCall), the placeHolder values(fv.Captures) are used to update the execution context, ensuring accurate and consistent results across iterations.

[ltzmaxwell]

also with line & column issue, neither is correct. to be fixed later.

[jaekwon]

Thank you Maxwell for the main readme description and for completing this PR.

[ltzmaxwell]

temp method to NOT support loopvar of range, to make it consistent with for i:=0;i<3;i++.

[ltzmaxwell]

also step on type decl.

[ltzmaxwell]

This PR is ready for review, cc: @jaekwon @thehowl @mvertes @deelawn @leohhhn . 🙏

[thehowl]

nitpicks for now; will need some more time to go through it all.

[thehowl]

Suggested change

	if hasName(loopfindr, dep) {
	if slices.Contains(loopfindr, dep) {

and remove hasName

[thehowl]

no, as there may only be 1 label; I think SetLabel is good.

[thehowl]

Please make a hiddenResultVariable and a isHiddenResultVariable function; so we don't have "magic" ".res_" strings in our code.

[thehowl]

Suggested change

	if attr.data == nil {
	if debug && attr.data == nil {

[thehowl]

Suggested change

	var bn BlockNode = sb.GetSource(store)
	bn := sb.GetSource(store)

[thehowl]

Can you replace the code in StaticBlock.GetStaticTypeOfAt with this method?

[thehowl]

Suggested change

	// NOTE: Extern names may also be local, if declared later.
	// NOTE: Extern names may also be local, if declared after usage as an extern
	// (thus shadowing the extern name).

[thehowl]

Suggested change

	// NOTE: if a name is later declared in this block later, it is both an
	// extern name with depth > 1, as well as local name with depth == 1.
	// NOTE: externs may also be shadowed later in the block. Thus, usages
	// before the declaration will have depth > 1; following it, depth == 1,
	// matching the two different identifiers they refer to.

[deelawn]

The solution seems to work well. Please see the comments and questions I left.

[deelawn]

This can be simplified to return ok

[deelawn]

In what case would something already be locally defined?

[deelawn]

Let's just make this if !isLocallyDefined2 and remove the if isLocallyDefined2 condition.

[deelawn]

Can this happen or not? If not, this conditional block can be replaced with lastLine = line to make it easier to understand what it's doing.

[deelawn]

Can we define .res_ as a constant?

[deelawn]

If we only do something when the name is in ATTR_LOOP_USES, can we get rid of the ATTR_LOOP_DEFINES lookup?

[deelawn]

What is this doing? ptr.TV's value is being set but is not used to construct the value being returned.

[deelawn]

Is this necessary? These captures are for values defined within the scope of a loop, right? It seems like the only way this value would be saved is if a reference was made to it during another assignment. In that case would this case inside this function handle it?

case *HeapItemValue:
		more = getSelfOrChildObjects(cv.Value.V, more)