Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ensure hashmap init clears maps #794

Merged
merged 2 commits into from
Jul 30, 2024
Merged

Conversation

tjungblu
Copy link
Contributor

@tjungblu tjungblu commented Jul 9, 2024

This reorders some statements in the hashmap initialization to ensure we always start fresh, even when no pageids were passed to it.

fixes #791

--

Might be better to fix this before #775? would make an easier backport.

@tjungblu tjungblu changed the title ensure hashmap init reindex ensure hashmap init clears maps Jul 9, 2024
freelist_test.go Outdated
f := newTestFreelist()
f.readIDs([]common.Pgid{5, 6, 8})

p := (*common.Page)(unsafe.Pointer(&buf[0]))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
p := (*common.Page)(unsafe.Pointer(&buf[0]))
p := common.LoadPage(buf)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied

freelist_test.go Outdated
@@ -179,6 +179,29 @@ func TestFreelist_releaseRange(t *testing.T) {
}
}

func TestFreeList_reload_page_dedupe(t *testing.T) {
var buf [4096]byte
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
var buf [4096]byte
buf := make([]byte, 4096)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

applied

freelist_test.go Outdated
Comment on lines 193 to 205
f2.free(common.Txid(5), common.NewPage(5, common.LeafPageFlag, 0, 4))
// reload should deduplicate as a pending page when reading from p's freelist
f2.reload(p)

if len(f2.getFreePageIDs()) != 0 {
t.Fatalf("expected empty; got=%v", f2.getFreePageIDs())
}
if exp := []common.Pgid{5, 6, 7, 8, 9}; !reflect.DeepEqual(exp, f2.pending[5].ids) {
t.Fatalf("exp=%v; got=%v", exp, f2.pending[5].ids)
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test is a little misleading.

  • If you reload (line 195) freelist, then you should also init/readIDs the freelist (f2) in the first place.
  • In real production use case, the only case to call reload if when we need to rollback a writing TXN. But in this case, you do not rollback the freelist before calling reload.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added your first suggestion. As for the second, we should hide the reload behind a Rollback as another refactoring step for #789

As of today, nothing stops you from reloading without the rollback.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Happy to hear other suggestions on how to replicate that inconsistent behavior though, feel free to send an alternative unit test.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As of today, nothing stops you from reloading without the rollback.

It means the interface isn't well designed. We need to continue to refactor the interface from users perspective. Will think about it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've rebased it against the refactoring now, let me know whether this is worth fixing or not. Otherwise we can close.

f2.Free(common.Txid(5), common.NewPage(5, common.LeafPageFlag, 0, 4))

// reload should deduplicate as a pending page when reading from p's freelist
f2.Reload(p)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you reload the freelist from a page, it means you already read the freelist from the page in the first place.

@ahrtr
Copy link
Member

ahrtr commented Jul 22, 2024

Please see the proposed unit test cases. We can revisit them when we update the interface later.


func TestFreeList_init(t *testing.T) {
	buf := make([]byte, 4096)
	f := newTestFreelist()
	f.Init(common.Pgids{5, 6, 8})

	p := common.LoadPage(buf)
	f.Write(p)

	f2 := newTestFreelist()
	f2.Read(p)
	require.Equal(t, common.Pgids{5, 6, 8}, f2.freePageIds())

	// When initializing the freelist with an empty list of page ID,
	// it should reset the freelist page IDs.
	f2.Init([]common.Pgid{})
	require.Equal(t, common.Pgids{}, f2.freePageIds())
}

func TestFreeList_reload(t *testing.T) {
	buf := make([]byte, 4096)
	f := newTestFreelist()
	f.Init(common.Pgids{5, 6, 8})

	p := common.LoadPage(buf)
	f.Write(p)

	f2 := newTestFreelist()
	f2.Read(p)
	require.Equal(t, common.Pgids{5, 6, 8}, f2.freePageIds())

	f2.Free(common.Txid(5), common.NewPage(10, common.LeafPageFlag, 0, 2))

	// reload shouldn't affect the pending list
	f2.Reload(p)

	require.Equal(t, common.Pgids{5, 6, 8}, f2.freePageIds())
	require.Equal(t, []common.Pgid{10, 11, 12}, f2.pendingPageIds()[5].ids)
}

@tjungblu
Copy link
Contributor Author

Works for me, there's still some nil/empty inconsistency:

       	Error:      	Not equal: 
        	            	expected: common.Pgids{}
        	            	actual  : common.Pgids(nil)
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,3 +1,2 @@
        	            	-(common.Pgids) {
        	            	-}
        	            	+(common.Pgids) <nil>

similar to here, I had to fix it with:
https://github.com/etcd-io/bbolt/pull/786/files#diff-1661bc3663fd3a2f6e7ebc479e8084bd0b1553a456792bdb0d005dbabee969f6R120

@tjungblu tjungblu force-pushed the 791_inconsistency branch 2 times, most recently from 5801f43 to 23807ad Compare July 22, 2024 13:59
This reorders some statements in the hashmap initialization to ensure we
always start fresh, even when no pageids were passed to it.

fixes etcd-io#791

Signed-off-by: Thomas Jungblut <[email protected]>
@tjungblu
Copy link
Contributor Author

okay that required a bit more empty/nil changes unfortunately. I'll leave those as a separated commit now....

This also rectifies a bunch of nil/empty differences between the
implementation that show up during init and page releases.

Signed-off-by: Thomas Jungblut <[email protected]>
Copy link
Member

@ahrtr ahrtr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

cc @fuweid

@k8s-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahrtr, tjungblu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ahrtr ahrtr merged commit 49c7697 into etcd-io:main Jul 30, 2024
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Development

Successfully merging this pull request may close these issues.

freelist hashmap/array inconsistency on page reload
3 participants