Generate sync webhook payloads in the dataloaders #16086

fowczarek · 2024-06-05T12:37:14Z

I want to merge this change because generating sync webhook payloads in the dataloaders.

TODO:

Adjust flow to handle multiple webhooks flows
create pre-payloads generated dataloader
Use created dataloader in all checkout resolvers with price recalcualtion
Cover flow with prices entered with taxes

In separate PR.

Cover draft orders flow.
Cover flow for other sync webhooks called via resolvers

Impact

New migrations
New/Updated API fields or mutations
Deprecated API fields or mutations
Removed API types, fields, or mutations

Docs

Link to documentation:

Pull Request Checklist

Privileged queries and mutations are either absent or guarded by proper permission checks
Database queries are optimized and the number of queries is constant
Database migrations are either absent or optimized for zero downtime
The changes are covered by test cases
All new fields/inputs/mutations have proper labels added (ADDED_IN_X, PREVIEW_FEATURE, etc.)
All migrations have proper dependencies
All indexes are added concurrently in migrations
All RunSql and RunPython migrations have revert option defined

saleor/webhook/transport/synchronous/transport.py

szczecha

I tested this PR with the Avatax app and different tax configurations.

The same query that causes the recursion error (get 100 checkouts with totalPrice while AvaTax App is configured) is no longer causing a recursion error. ❤️

I have explored the checkout and webhooks areas to ensure that nothing breaks.

Unfortunately, it doesn't solve the other issue of how we can observe recursion error (Taxes are 0 when calling updateDeliveryMethod while the AvaTax App is configured and there is a custom app with a checkout updated webhook with the totalPrice field). Still, this wasn't the main goal of this fix.

db-queries · 2024-07-17T08:46:45Z

Here is the report for f41ff55 (saleor:PE-282-generate-sync-payload-in-dataloaders)
Base comparison is c908c33.

Found 21 differences! (click me)

# saleor.graphql.accountbenchmark account
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  addresses for federation query count                                    	          9	          9	              2
  customers query                                                         	         51	         51	              0
  delete staff members                                                    	         40	         40	              0
  query staff user                                                        	         22	         22	              3
  staff create                                                            	         22	         22	              3
  staff update groups and permissions                                     	         35	         35	              5
  users for federation query count                                        	          8	          8	              3

# saleor.graphql.accountbenchmark permission group
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  groups for federation query count                                       	          3	          3	              0
  permission group create                                                 	         24	         24	              0
  permission group delete                                                 	         29	         29	              3
  permission group query                                                  	         14	         14	              0
  permission group update                                                 	         43	         43	              2
  permission group update remove users with manage staff                  	         35	         35	              3

# saleor.graphql.appbenchmarks app extensions
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  app extensions                                                          	         25	         25	             10
  app extensions with filter[filter0]                                     	         24	         24	             10
  app extensions with filter[filter1]                                     	         12	         12	              2
  app extensions with filter[filter2]                                     	         24	         24	             10
  app extensions with filter[filter3]                                     	         12	         12	              2

# saleor.graphql.appbenchmarks apps
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  apps for federation query count                                         	          9	          9	              3
  apps with tokens and webhooks                                           	          9	          9	              0

# saleor.graphql.attributebenchmark attribute
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  attribute translation                                                   	          6	          6	              0
  attribute value translation                                             	         20	         20	              0
  query attribute                                                         	          6	          6	              0
  query attributes                                                        	          8	          8	              0

# saleor.graphql.channelbenchmark channel
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  channels query                                                          	          5	          5	              0

# saleor.graphql.channelmutations channel update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  channel update mutation remove warehouse                                	         39	         39	              6

# saleor.graphql.checkoutbenchmark checkout mutations
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
- add billing address to checkout                                         	         65	         67	              7
- add checkout lines                                                      	         91	         93	             16
- add checkout lines catalogue discount applies                           	        189	        191	             16
- add checkout lines gift discount applies                                	        153	        155	             24
- add checkout lines multiple catalogue discount applies                  	        204	        206	             21
- add checkout lines order discount applies                               	        167	        169	             19
- add checkout lines with external shipping                               	        174	        176	             72
- add checkout lines with reservations                                    	        191	        195	             71
- add delivery to checkout                                                	         82	         84	             14
- add shipping to checkout                                                	         83	         85	             14
- checkout email update                                                   	         44	         46	              3
  checkout payment charge                                                 	         60	         60	             17
- checkout shipping address update                                        	         77	         79	              8
- checkout voucher code                                                   	         88	         90	             13
  complete checkout                                                       	        246	        246	             82
  complete checkout preorder                                              	        241	        241	             83
  complete checkout with digital line                                     	        284	        284	             98
  complete checkout with out of stock webhook                             	        248	        248	             82
  complete checkout with single line                                      	        248	        248	             82
- create checkout                                                         	         69	         71	              5
- create checkout with gift promotion                                     	        106	        108	             14
- create checkout with order promotion                                    	         91	         93	              5
- create checkout with reservations                                       	        157	        161	             31
  customer complete checkout                                              	        259	        259	             88
  customer complete checkout for cc                                       	        234	        234	             78
- update checkout lines                                                   	         84	         86	             15
- update checkout lines with reservations                                 	        197	        201	             82

# saleor.graphql.checkoutbenchmark checkouts
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  staff multiple checkouts                                                	          6	          6	              0

# saleor.graphql.checkoutbenchmark homepage
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
- user checkout details                                                   	         55	         57	              5
- user checkout details with tax app                                      	         81	         86	             30

# saleor.graphql.discountbenchmark promotion create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion create                                                        	         23	         23	              1
  promotion create order promotion                                        	         37	         37	              7

# saleor.graphql.discountbenchmark promotion delete
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  gift promotion delete                                                   	         25	         25	              1
  promotion delete                                                        	         30	         30	              1

# saleor.graphql.discountbenchmark promotion rule create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion rule create                                                   	         23	         23	              1
  promotion rule create gift                                              	        134	        134	              7

# saleor.graphql.discountbenchmark promotion rule delete
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion rule delete                                                   	         23	         23	              0

# saleor.graphql.discountbenchmark promotion rule update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion rule update                                                   	         38	         38	              0

# saleor.graphql.discountbenchmark promotion update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotion update                                                        	         16	         16	              0

# saleor.graphql.discountbenchmark promotions
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  promotions querytest promotions query                                   	          9	          9	              0

# saleor.graphql.discountbenchmark sales
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  sales query with channel slug                                           	          9	          9	              0
  sales query without channel slug                                        	          7	          7	              0

# saleor.graphql.discountbenchmark voucher code bulk delete
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  voucher code bulk delete queries                                        	         20	         20	              2

# saleor.graphql.discountbenchmark vouchers
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  vouchers query with channel slug                                        	         20	         20	              0
  vouchers query withot channel slug                                      	         19	         19	              0

# saleor.graphql.giftcardbenchmark gift card mutations
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  create never expiry gift card                                           	         27	         27	              4
  gift card bulk activate by staff                                        	         15	         15	              3
  update gift card                                                        	         35	         35	              4

# saleor.graphql.giftcardbenchmark gift card queries
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  filter gift cards by products                                           	         14	         14	              3
  filter gift cards by tags                                               	         12	         12	              3
  filter gift cards by used by user                                       	         13	         13	              3
  query gift card details                                                 	         14	         14	              3
  query gift cards                                                        	         12	         12	              3

# saleor.graphql.orderbenchmark fulfillment
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  fulfillment query                                                       	          6	          6	              0

# saleor.graphql.orderbenchmark fulfillment refund and return products
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  fulfillment refund products order lines                                 	         61	         61	              0
  fulfillment return products order lines                                 	        103	        103	              7

# saleor.graphql.orderbenchmark order
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  staff multiple draft orders                                             	          7	          7	              0
  staff multiple orders                                                   	          7	          7	              0
  staff order details                                                     	         57	         57	              7
  user order details                                                      	         54	         54	              8

# saleor.graphql.orderbenchmark order bulk create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  order bulk create                                                       	         20	         20	              2

# saleor.graphql.orderbenchmark order fulfill
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  order fulfill                                                           	         18	         18	              0
  order fulfill with gift cards                                           	         18	         18	              0

# saleor.graphql.orderbenchmark order lines create
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  order lines create                                                      	         95	         95	             17
  order lines create variants on promotion                                	        154	        154	             22

# saleor.graphql.pagebenchmark page type
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  page types for federation query count                                   	          3	          3	              0
  query page type                                                         	         18	         18	              6
  query page types                                                        	         25	         25	             12

# saleor.graphql.paymentbenchmark payment transactions
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  payment transactions                                                    	         54	         54	             20

# saleor.graphql.productbenchmark category
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  categories children                                                     	          4	          4	              0
  categories for federation query count                                   	          3	          3	              0
  category delete                                                         	         48	         48	              4
  category view                                                           	         28	         28	              1

# saleor.graphql.productbenchmark collection
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  collection add products                                                 	         22	         22	              5
  collection bulk delete                                                  	         27	         27	              3
  collection view                                                         	          7	          7	              0
  collections for federation query count                                  	          4	          4	              0
  create collection                                                       	         37	         37	              5
  delete collection                                                       	         40	         40	              5
  remove products from collection                                         	         19	         19	              5
  retrieve collection channel listings                                    	          5	          5	              0

# saleor.graphql.productbenchmark homepage
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  report product sales                                                    	         10	         10	              3
  retrieve product list                                                   	          4	          4	              0

# saleor.graphql.productbenchmark product
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  filter products by attributes                                           	          7	          7	              0
  filter products by boolean attributes                                   	         14	         14	              0
  filter products by gift card                                            	         12	         12	              1
  filter products by numeric attributes                                   	         13	         13	              0
  product create                                                          	         48	         48	              3
  product details                                                         	         41	         41	              1
  product translations                                                    	          5	          5	              0
  products for federation query count                                     	          6	          6	              1
  products media for federation query count                               	          5	          5	              0
  products types for federation query count                               	          2	          2	              0
  retrieve channel listings                                               	         21	         21	              0
  retrieve product attributes                                             	          8	          8	              0
  retrieve product images                                                 	          4	          4	              0
  retrieve product media                                                  	          4	          4	              0
  retrive products with product types and attributes                      	          7	          7	              0
  update product                                                          	         71	         71	              5

# saleor.graphql.productbenchmark product variant channel listing update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  variant channel listing update                                          	         35	         35	              3

# saleor.graphql.productbenchmark variant
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  product variant bulk create                                             	         45	         45	              1
  product variant create                                                  	         61	         61	              3
  products variants for federation query count                            	          6	          6	              1
  retrieve variant list                                                   	         34	         34	              1
  update product variant                                                  	         64	         64	              7

# saleor.graphql.productbenchmark variant stocks
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  product variants stocks create                                          	         28	         28	              4
  product variants stocks create with single webhook called               	         27	         27	              4
  product variants stocks delete by id                                    	         29	         29	              4
  product variants stocks delete by sku                                   	         29	         29	              4
  product variants stocks delete with out of stock webhook many calls     	         25	         25	              3
  product variants stocks update by sku                                   	         27	         27	              4
  product variants stocks update byid                                     	         27	         27	              4
  query product variants stocks                                           	         10	         10	              0

# saleor.graphql.producttest product sorting attributes
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  sort product not having attribute data                                  	         20	         20	              0

# saleor.graphql.shippingbenchmark shipping methods
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  vouchers query with channel slug                                        	          7	          7	              0
  vouchers query without channel slug                                     	          7	          7	              0

# saleor.graphql.shippingbenchmark shipping zones
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  shipping zones query                                                    	          6	          6	              0

# saleor.graphql.shopbenchmark homepage
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  retrieve shop                                                           	          2	          2	              0

# saleor.graphql.warehousebenchmark stock bulk update
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  stocks bulk update queries count                                        	         31	         31	              5

# saleor.graphql.warehousebenchmark stocks
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  stocks query                                                            	          5	          5	              0

# saleor.graphql.webhookbenchmark webhook events
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  webhooks                                                                	          6	          6	              0

# saleor.orderbenchmark fetch order prices
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  fetch order prices catalogue discount                                   	         51	         51	              7
  fetch order prices multiple catalogue discounts                         	        113	        113	             21

# saleor.ordertest fetch
  test name                                                               	left count 	right count	duplicate count
  ------------------------------------------------------------------------	-----------	-----------	---------------
  fetch draft order lines info                                            	         21	         21	              1

codecov · 2024-07-17T08:47:11Z

Codecov Report

Attention: Patch coverage is 91.66667% with 10 lines in your changes missing coverage. Please review.

Project coverage is 94.11%. Comparing base (b81acac) to head (f41ff55).
Report is 2 commits behind head on main.

Files	Patch %	Lines
saleor/graphql/webhook/subscription_payload.py	71.42%	5 Missing and 3 partials ⚠️
saleor/graphql/webhook/dataloaders.py	96.07%	0 Missing and 2 partials ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main   #16086    +/-   ##
========================================
  Coverage   94.10%   94.11%            
========================================
  Files         937      938     +1     
  Lines       73081    73181   +100     
  Branches     8654     8671    +17     
========================================
+ Hits        68776    68871    +95     
+ Misses       2739     2738     -1     
- Partials     1566     1572     +6

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

saleor/graphql/webhook/subscription_payload.py

kadewu · 2024-07-18T15:14:51Z

saleor/graphql/webhook/subscription_payload.py

+ results, app_id=app_id, subscription_query=subscription_query
+ ) -> Union[Promise[Optional[dict[str, Any]]], Optional[dict[str, Any]]]:
+ if hasattr(results, "errors"):
+ logger.warning(


Just noting the codecov complains here.

fowczarek added skip changelog main labels Jun 5, 2024

fowczarek self-assigned this Jun 5, 2024

github-advanced-security bot found potential problems Jun 14, 2024

View reviewed changes

saleor/webhook/transport/synchronous/transport.py Fixed Show fixed Hide fixed

fowczarek force-pushed the PE-282-generate-sync-payload-in-dataloaders branch 2 times, most recently from 8e92da7 to 6bdd611 Compare July 5, 2024 10:59

fowczarek force-pushed the PE-282-generate-sync-payload-in-dataloaders branch from a770d48 to dba54d1 Compare July 11, 2024 13:46

szczecha added the test deployment Deploy test environment for pull request label Jul 12, 2024

github-actions bot temporarily deployed to pe-282-generate-sync-payload-in-dataloaders July 12, 2024 11:06 Inactive

szczecha previously approved these changes Jul 15, 2024

View reviewed changes

fowczarek dismissed szczecha’s stale review via a165f62 July 16, 2024 10:36

fowczarek force-pushed the PE-282-generate-sync-payload-in-dataloaders branch from dba54d1 to a165f62 Compare July 16, 2024 10:36

github-actions bot temporarily deployed to pe-282-generate-sync-payload-in-dataloaders July 16, 2024 10:37 Inactive

fowczarek force-pushed the PE-282-generate-sync-payload-in-dataloaders branch from a165f62 to 2f9b6f1 Compare July 17, 2024 08:25

github-actions bot temporarily deployed to pe-282-generate-sync-payload-in-dataloaders July 17, 2024 08:26 Inactive

fowczarek removed the test deployment Deploy test environment for pull request label Jul 17, 2024

fowczarek requested review from a team July 17, 2024 09:00

fowczarek marked this pull request as ready for review July 17, 2024 09:46

IKarbowiak reviewed Jul 17, 2024

View reviewed changes

saleor/graphql/webhook/subscription_payload.py Show resolved Hide resolved

saleor/graphql/webhook/subscription_payload.py Outdated Show resolved Hide resolved

fowczarek added 8 commits July 17, 2024 15:25

[POC] Pass paylaod from resolver to webhook plugin

4c7b976

Adjust flow to handle multiple webhooks flows

f2ffaa7

Add PregeneratedSubscriptionPayloadsByCheckoutTokenLoader dataloader

d7c0155

[POC] Generate payload as Promice

0ad3ec4

Add unit tests

e77c45c

Handle pregenerated paylods in checkout subtotal resolver

ee2a41b

Handle pregenerated paylods in checkout total balance resolver

8cf39c5

Add workaround for missing using in get delivery method info

c1cc84c

fowczarek added 9 commits July 17, 2024 15:25

Handle pregenerated paylods in checkout shipping price resolver

4649e05

Handle pregenerated paylods in checkout authorize status resolver

b068bc3

Handle pregenerated paylods in checkout charge status resolver

73aab77

Refactor get_dataloaders_for_fetching_checkout_data

1cf22b1

Handle pregenerated paylods in checkout lines unit price resolver

1f7d561

Handle pregenerated paylods in checkout lines total price resolver

b30f518

Fix docstring in create_delivery_for_subscription_sync_event

58ea671

Refactor PregeneratedCheckoutTaxPayloadsByCheckoutTokenLoader

0da9d83

Fix reveiw remarks

f41ff55

fowczarek force-pushed the PE-282-generate-sync-payload-in-dataloaders branch from 2f9b6f1 to f41ff55 Compare July 18, 2024 07:28

fowczarek requested a review from IKarbowiak July 18, 2024 07:50

IKarbowiak approved these changes Jul 18, 2024

View reviewed changes

IKarbowiak requested a review from a team July 18, 2024 08:47

kadewu approved these changes Jul 18, 2024

View reviewed changes

kadewu requested review from a team July 18, 2024 15:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Generate sync webhook payloads in the dataloaders #16086

Generate sync webhook payloads in the dataloaders #16086

fowczarek commented Jun 5, 2024 •

edited

Loading

szczecha left a comment •

edited

Loading

db-queries bot commented Jul 17, 2024 •

edited

Loading

codecov bot commented Jul 17, 2024 •

edited

Loading

kadewu Jul 18, 2024

Generate sync webhook payloads in the dataloaders #16086

Are you sure you want to change the base?

Generate sync webhook payloads in the dataloaders #16086

Conversation

fowczarek commented Jun 5, 2024 • edited Loading

Impact

Docs

Pull Request Checklist

szczecha left a comment • edited Loading

Choose a reason for hiding this comment

db-queries bot commented Jul 17, 2024 • edited Loading

codecov bot commented Jul 17, 2024 • edited Loading

Codecov Report

kadewu Jul 18, 2024

Choose a reason for hiding this comment

fowczarek commented Jun 5, 2024 •

edited

Loading

szczecha left a comment •

edited

Loading

db-queries bot commented Jul 17, 2024 •

edited

Loading

codecov bot commented Jul 17, 2024 •

edited

Loading