[refactor] serdes pack/deserialize APIs #12524

smackesey · 2023-02-24T14:00:01Z

Summary & Motivation

Consolidate APIs of dagster._serdes and update all callsites to use the new (typed) APIs.

Major change is the consolidation of the various deserialize/serialize/pack/unpack functions used throughout the codebase into just 4 module-public functions, which incorporate typing:

deserialize_value
serialize_value
unpack_value
pack_value

*_value was used as opposed to bare e.g. deserialize because serialize_value and deserialize_value are part of our public API. This PR does not break the existing contract of those functions but does add some optional args to them.

There is a significant companion PR for internal here: https://github.com/dagster-io/internal/pull/4974

How I Tested These Changes

BK

vercel · 2023-02-24T14:00:06Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

2 Ignored Deployments

Name	Status	Preview	Comments	Updated
dagit-storybook	⬜️ Ignored (Inspect)			Mar 9, 2023 at 11:18PM (UTC)
dagster	⬜️ Ignored (Inspect)			Mar 9, 2023 at 11:18PM (UTC)

smackesey · 2023-02-24T14:00:09Z

Current dependencies on/for this PR:

master
- PR Remove deprecated MetadataEntry constructors, update entry_data internal refs #12724
  - PR remove MetadataEntry APIs #12769
  - PR MetadataEntryUnion #12725
    - PR Remove PartitionMetadataEntry #12726
      - PR [typing/runtime] serdes #12523
        
        PR [refactor] serdes pack/deserialize APIs #12524 👈
        
        PR [refactor] MayHaveInstanceWeakRef error on access non-existent _instance #12553
        
        PR [typing/static] telemetry #12714
        
        PR [typing/static] daemon/scheduler/instance #12715
        
        PR [refactor] ConfigurableClass APIs #12716
      - PR [refactor] Update internal representation of metadata to always use metadata dict #12770

This comment was auto-generated by Graphite.

alangenfeld

this PR seems to be in a weird stacked state where it contains additional changes - but i think i covered it well enough to believe this is good. Good tidying up on the API.

alangenfeld · 2023-03-08T17:33:49Z

python_modules/dagster/dagster/_core/execution/plan/execute_step.py

@@ -497,28 +496,6 @@ def _get_output_asset_materializations(
        tags[BACKFILL_ID_TAG] = backfill_id

    if asset_partitions:
-        metadata_mapping: Dict[


are these metadata changes supposed to be in this PR or is this a stacking artifact?

Stacking artifact, this happens in between time when downstack PR lands and target branch gets rebased.

[INTERNAL_BRANCH=sean/refactor-serdes]

This was referenced Feb 24, 2023

[typing/static] serdes #12522

Merged

[typing/runtime] serdes #12523

Merged

smackesey force-pushed the sean/typing-runtime-serdes branch from 7a55dd1 to bf6b55d Compare February 24, 2023 14:49

smackesey force-pushed the sean/refactor-serdes branch from 2102cc0 to 7ad4c5c Compare February 24, 2023 14:49

smackesey force-pushed the sean/typing-runtime-serdes branch from bf6b55d to 94206fe Compare February 24, 2023 15:03

smackesey force-pushed the sean/refactor-serdes branch from 7ad4c5c to 9b3d7e3 Compare February 24, 2023 15:04

smackesey force-pushed the sean/typing-runtime-serdes branch from 94206fe to c1faecb Compare February 24, 2023 15:30

smackesey force-pushed the sean/refactor-serdes branch from 9b3d7e3 to b26d635 Compare February 24, 2023 15:30

smackesey force-pushed the sean/typing-runtime-serdes branch 2 times, most recently from f005375 to 5dd6dfe Compare February 24, 2023 17:02

smackesey force-pushed the sean/refactor-serdes branch from b26d635 to d62cc09 Compare February 24, 2023 17:02

smackesey force-pushed the sean/typing-runtime-serdes branch from 5dd6dfe to 902edc3 Compare February 26, 2023 15:30

smackesey force-pushed the sean/refactor-serdes branch 2 times, most recently from b403d5e to 0430200 Compare February 26, 2023 15:41

smackesey mentioned this pull request Feb 26, 2023

[refactor] MayHaveInstanceWeakRef error on access non-existent _instance #12553

Merged

smackesey force-pushed the sean/typing-runtime-serdes branch from 902edc3 to a9d9989 Compare March 4, 2023 15:00

smackesey force-pushed the sean/refactor-serdes branch from 0430200 to 317e153 Compare March 4, 2023 15:00

This was referenced Mar 4, 2023

[refactor] Refactor of serdes serializers #12713

Merged

[typing/static] telemetry #12714

Merged

[typing/static] daemon/scheduler/instance #12715

Merged

[refactor] ConfigurableClass APIs #12716

Merged

smackesey force-pushed the sean/typing-runtime-serdes branch from a9d9989 to f1f97a6 Compare March 4, 2023 15:57

smackesey force-pushed the sean/refactor-serdes branch from 317e153 to db7330e Compare March 4, 2023 16:19

smackesey force-pushed the sean/typing-runtime-serdes branch from f1f97a6 to 557c7ef Compare March 4, 2023 19:18

smackesey force-pushed the sean/refactor-serdes branch 2 times, most recently from 7924750 to 378dbbc Compare March 4, 2023 20:26

smackesey force-pushed the sean/typing-runtime-serdes branch from 557c7ef to 399da44 Compare March 5, 2023 22:49

smackesey force-pushed the sean/refactor-serdes branch from 378dbbc to d7f8955 Compare March 5, 2023 22:49

smackesey force-pushed the sean/typing-runtime-serdes branch from 8bbc961 to 82a786b Compare March 7, 2023 18:12

smackesey force-pushed the sean/refactor-serdes branch from 5bbaade to e285623 Compare March 7, 2023 18:12

smackesey force-pushed the sean/typing-runtime-serdes branch from 82a786b to 1f0b504 Compare March 7, 2023 20:47

smackesey force-pushed the sean/refactor-serdes branch from e285623 to f7514d0 Compare March 7, 2023 20:47

smackesey force-pushed the sean/typing-runtime-serdes branch from 1f0b504 to 6351d37 Compare March 7, 2023 21:46

smackesey force-pushed the sean/refactor-serdes branch from f7514d0 to 169a425 Compare March 7, 2023 21:46

smackesey force-pushed the sean/typing-runtime-serdes branch from 6351d37 to 4dc724a Compare March 7, 2023 22:38

smackesey force-pushed the sean/refactor-serdes branch from 169a425 to 14b2087 Compare March 7, 2023 22:38

smackesey force-pushed the sean/typing-runtime-serdes branch from 4dc724a to cd0f6c9 Compare March 8, 2023 01:19

smackesey force-pushed the sean/refactor-serdes branch from 14b2087 to 653b90b Compare March 8, 2023 01:19

smackesey force-pushed the sean/typing-runtime-serdes branch from cd0f6c9 to 50dea09 Compare March 8, 2023 10:48

smackesey force-pushed the sean/refactor-serdes branch 2 times, most recently from 5bcadd0 to ebb24e3 Compare March 8, 2023 11:38

smackesey force-pushed the sean/typing-runtime-serdes branch from 50dea09 to 9d9770e Compare March 8, 2023 11:39

smackesey force-pushed the sean/refactor-serdes branch from ebb24e3 to 7f48b3e Compare March 8, 2023 11:39

smackesey changed the title ~~[refactor] serdes~~ [refactor] serdes pack/deserialize APIs Mar 8, 2023

smackesey force-pushed the sean/typing-runtime-serdes branch from 9d9770e to bf86440 Compare March 8, 2023 17:03

alangenfeld approved these changes Mar 8, 2023

View reviewed changes

smackesey force-pushed the sean/typing-runtime-serdes branch from bf86440 to 6807c35 Compare March 8, 2023 19:01

smackesey force-pushed the sean/refactor-serdes branch from 7f48b3e to 3363415 Compare March 8, 2023 19:01

smackesey force-pushed the sean/typing-runtime-serdes branch from 6807c35 to c80abcb Compare March 8, 2023 21:13

smackesey force-pushed the sean/refactor-serdes branch from 3363415 to 274bb17 Compare March 8, 2023 21:13

Base automatically changed from sean/typing-runtime-serdes to master March 8, 2023 21:40

smackesey force-pushed the sean/refactor-serdes branch 3 times, most recently from 3eb8585 to a8ac04b Compare March 9, 2023 22:55

Convert untyped deserialize calls to deserialize_as

de4b416

[INTERNAL_BRANCH=sean/refactor-serdes]

smackesey force-pushed the sean/refactor-serdes branch from a8ac04b to de4b416 Compare March 9, 2023 23:17

smackesey merged commit b04cb79 into master Mar 10, 2023

smackesey deleted the sean/refactor-serdes branch March 10, 2023 00:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[refactor] serdes pack/deserialize APIs #12524

[refactor] serdes pack/deserialize APIs #12524

smackesey commented Feb 24, 2023 •

edited

Loading

vercel bot commented Feb 24, 2023 •

edited

Loading

smackesey commented Feb 24, 2023 •

edited

Loading

alangenfeld left a comment

alangenfeld Mar 8, 2023

smackesey Mar 8, 2023

[refactor] serdes pack/deserialize APIs #12524

[refactor] serdes pack/deserialize APIs #12524

Conversation

smackesey commented Feb 24, 2023 • edited Loading

Summary & Motivation

How I Tested These Changes

vercel bot commented Feb 24, 2023 • edited Loading

smackesey commented Feb 24, 2023 • edited Loading

alangenfeld left a comment

Choose a reason for hiding this comment

alangenfeld Mar 8, 2023

Choose a reason for hiding this comment

smackesey Mar 8, 2023

Choose a reason for hiding this comment

smackesey commented Feb 24, 2023 •

edited

Loading

vercel bot commented Feb 24, 2023 •

edited

Loading

smackesey commented Feb 24, 2023 •

edited

Loading