support value in MaterializeResult

[jamiedemaria]

What's the use case?

With the introduction of MaterializeResult, we anticipate that the ability to store scalar values to access in downstream assets will be a request feature.

Example use case:

@asset 
def table_1():
     full_table_name = "my_schema.table_1"
     snowflake.query(f"CREATE TABLE {full_table_name}  ...")
    return MaterializeResult(value=full_table_name)

@asset(
    deps=[table_1]
)
def table_2(context):
     table_1_table_name = context.get_materialize_result_value(table_1)  # some API for getting the value from MaterializeResult
     snowflake.query(f"CREATE TABLE my_schema.table_2 AS SELECT * FROM {table_1_table_name} ...")

Some things to consider:

• How will this value be stored? Will we just store the value or the entire MaterializeResult?
• If the user has a default IO manager set that can't handle storing scalars or full MaterializeResult objects (for example, the DB io managers can only store tabular data), how should we handle that? MaterializeResults are supposed to be a way out of I/O manager world, so needing to do this

@asset(
    io_manager_key="file_system"
)
def table_1():
     full_table_name = "my_schema.table_1"
     snowflake.query(f"CREATE TABLE {full_table_name}  ...")
    return MaterializeResult(value=full_table_name)

would go against the purpose of MaterializeResult in the first place.

Ideas of implementation

1. instead of converting MaterializeResult into Output(None), convert to Output(MaterializeResult(...)) and store the full object via IO manager
2. extract the value from MaterializeResult and convert that to Output(materialize_result.value). Store that via the IO manager
   2a. As Sandy mentioned below, use the metadata system for storing the scalar instead. I'm liking this idea at the moment as it sidsteps the IO managers entirely, which stays more inline with the purpose of MaterializeResult

Additional information

No response

Message from the maintainers

Impacted by this issue? Give it a 👍! We factor engagement into prioritization.

[sryza]

I wonder if it makes sense to use the metadata system for this? At least in the case above, materialization metadata would work well, and would also have the advantage of being consumable via the UI.

[jamiedemaria]

Depending on the approach taken for storing value, we may also want to do something like this #16136 and bypass the IO manager when MaterializeResult is returned.

[yuhan]

I wonder if it makes sense to use the metadata system for this? At least in the case above, materialization metadata would work well, and would also have the advantage of being consumable via the UI.

@sryza did you mean #8521 could an alternative to enabling this?

[sryza]

@yuhan exactly

[yuhan]

with #20091, could we now expose upstream metadata through AssetExecutionContext now?