Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(bugfix) add build params to Memoize cache key #3292

Closed
wants to merge 1 commit into from
Closed

(bugfix) add build params to Memoize cache key #3292

wants to merge 1 commit into from

Conversation

andrii-kasparevych
Copy link

This PR fixes the problem described here in details.
In short, when caching the Starlark/JSONNET conversion results, the build parameters are ignored which is harmful when the parameters are used for pipeline generation.

@bradrydzewski
Copy link

bradrydzewski commented Nov 30, 2022

@andrii-kasparevych golang map iteration order is random which would result in different cache keys.

When iterating over a map with a range loop, the iteration order is not specified and is not guaranteed to be the same from one iteration to the next. If you require a stable iteration order you must maintain a separate data structure that specifies that order. From go.dev/blog/maps.

This could be solved by sorting to ensure deterministic ordering (snippet below, found on stackoverflow, needs testing). Alternatively you could set the DRONE_CONVERT_PLUGIN_CACHE_SIZE to 0 which disables caching. Generally speaking there should be no problem disabling caching unless you are executing 1000+ pipelines per hour.

func mapToHash(m map[string]string) string {
	keys := make([]string, len(m))
	h := sha256.New()
	i := 0
	for k := range m {
		keys[i] = k
		i++
	}
	sort.Strings(keys)
	for _, k := range keys {
		ioutil.WriteString(h, k)
		ioutil.WriteString(h, m[k])
	}

	return fmt.Sprintf("%x", h.Sum(nil))
}

@andrii-kasparevych
Copy link
Author

@bradrydzewski thanks for the hint. I think I have confused the 2 memoise.go files in different places: here and here.

I will try disabling the caching and see if that does the trick

@andrii-kasparevych
Copy link
Author

Disabling the caching worked for us, I will close the PR then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants