Implement get_var step

[vito]

Summary

This issue is for implementing concourse/rfcs#27 as described.

As the load_var step and concept of "local var sources" is already implemented, this issue tracks the delta:

• Implement the get_var step, which fetches a var from a var source and can trigger the job when the value changes from the previously used value.

Context

• RFC: RFC: var steps + local var sources rfcs#27

[vixus0]

What about maintaining traceability and reproducibility when using get_var to trigger jobs? With resources, triggering inputs to a job are very clear, and I can use version pinning to reproduce a job run. With get_var as currently proposed, it will be less obvious what can cause a job to be triggered automatically and it won't be possible to reproduce previous job runs as vars aren't versioned.

Spooky action at a distance :p

[vito]

@vixus0 What you describe is true, but those sounds like reasons to use a resource and not a var source. They aren't meant to have the same properties of being reproducible and versioned - they're for different things.

They're meant to be used for values which shouldn't be versioned - for example, I wouldn't want to re-trigger an old build and then have it fail because it used a credential that has since been rotated. Or if I re-trigger a build that's time-triggered and uses the time var value to record a deploy event, I would expect the event to be saved with the current time, not the original time, because that's when the deploy is happening. Or if I have a job that configures pipelines across all PRs using a var source to represent the set of PRs, when I re-trigger a build I would expect it to run with the current set of PRs, not whatever set it had at the time the build originally ran.

So the long and short of it is that you would use resources for things which are versioned and reproducible, and vars for things that only ever have a "current value" that you want to use.

Does this make sense? :)

Re: obviousness, I think this is kind of up in the air, since we don't really know how this feels to use yet. Once we get some feedback cycles going I think we can start to figure out the UI/UX.

[vixus0]

@vito Yeah I guess UI/UX is what I'm curious about. load_var makes perfect sense for the reasons you describe but triggering from a get_var feels like there should be some corresponding indicator on the pipeline. Maybe that's something that can come later though.

I'm asking these questions because when var source and resource prototypes become a thing (both would be useful for how I use Concourse) I can see a lot of overlap in behaviour, at least when it comes to triggering pipelines. It seems that versioning will be the sole distinguishing characteristic between var sources and resource types.

[vito]

@vixus0 Yeah, figuring out how this integrates into the existing pipeline UI will be important. I agree that there should be something there to indicate it as another input to the job. It would probably make sense to have the same trigger/non-trigger graph line UI just to build on the existing mental model, but I'm not sure how the graph nodes themselves should look or behave.

At the same time though we'll probably want to have a UI that conveys that they aren't the same as resources and try to make their differences seem obvious and complementary.

Thanks for the feedback! These are good questions, it helps me get an idea of how to properly frame these two concepts.

[xeivieni]

Hi @vito, I see that only time and vault are mentioned in the RFC for the get_var step. Are other sources considered ?
Could someone create a custom get_var source provider, based on existing resources for instance ?

[vito]

@xeivieni yep that's the plan - this is all tied to var sources, and we're going to have var sources able to be implemented by prototypes (concourse/rfcs#37)

[vito]

This is now in progress here: #6274

Larger epic: #5229

Epic project for tracking progress: https://project.concourse-ci.org/projects/MDc6UHJvamVjdDU5MTE2Mjc=