Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backend payment optimism #1195

Merged
merged 129 commits into from
Jul 1, 2024
Merged

backend payment optimism #1195

merged 129 commits into from
Jul 1, 2024

Conversation

huumn
Copy link
Member

@huumn huumn commented May 27, 2024

This is functionally done - all known problems, deficiencies have been completed afaik. All that's left is review, cleanup, and testing.

Fully ready to review and deploy.

Regression warning:

  • I've completely removed referral payments (attribution still works/happens) from this PR. Once this has been shipped to prod I'll begin working on referral overhaul #1144.

The most important parts of this PR:

  • api/paidAction - has all the paid actions in their own files
  • api/paidAction/index.js - drives all the paid actions determining whether to use fee credits, optimism, or pessimism
  • components/use-paid-mutation.js - wraps useMutation to handle responses of paid actions and pay any invoices returned and give usePaidMutation callers callbacks about payment progress
  • worker/paidAction.js - handles state transitions of paid actions and their invoices

The biggest concerns with the PR:

  • Is it easy to understand how paid actions work and how to implement them? ie DX
  • Is the UX of paid actions good, regardless of whether the action is optimistic, pessimistic, anon, paid with a qr, etc.
  • Is concurrency handled appropriately

To achieve an ideal non-custodial UX, we are doing what we're calling 2-stage optimism:

  1. frontend optimism: instantly reports success on the frontend while sending the requested action to the backend
  2. backend optimism: stores the requested action in a PENDING state, returning success, an invoice, and appearing as-if the payment is confirmed to the actor

We then transition the action state to PAID or FAILED depending on the payment state. When the payment succeeds, we run the action side effects and the action is visible to everyone. When the payment fails, we report the failure in the stacker's notifications where they can retry the payment.


Doing this requires an overhaul to all paid actions. The requirements on paid actions are summarized in the following table:

anonable & pessimistic optimistic qr payable p2p wrapped side effects fee credits payable
posts x x x x x
comments x x x x x
zaps x x x x x x
downzaps x x x x
poll votes x x x
territory actions x x
donations x x x
update posts x x x
update comments x x x

This pr intends to implement backend optimism for all actions. We aim to take a relatively declarative approach to creating paid actions, where paid actions implement a standard interface so that paid actions aren't directly concerned with managing the payments. So far, what this looks like is paid actions are implemented in a separate file in the paidAction folder with an interface that defines the following functionality:

  1. getCost - returns the cost of the action given the action's arguments so that an invoice can be created
  2. perform - inserts the action's records leaving them in a PENDING state with reference to the corresponding invoiceId
  3. onPaid - transitions the records from perform to a PAID state and does any side effects (e.g. denormalization)

A lot of care will need to be taken wrt atomicity of state transitions and the performance of making PENDING actions appear PAID selectively.

To avoid having to serialize state transitions, we are using optimistic concurrency control.


This PR is being completed in stages:

  • make all actions work with custodial funds
  • make all pessimistic actions work
  • make all optimistic actions work
  • modify all queries to filter out PENDING actions expect if the actor is viewing them
  • notifications for failed payments
  • fix all the missing things
    • todos scattered throughout
    • zap undos
    • delete/remindme bot toasts
    • territory unarchive and transfer?
  • ux of failed post/comment payments (perhaps where edit XX:XX usually is we can have payment status)
  • wot and materialized view consequences
  • prevent commenting on/zapping pending/failed items
  • retries (mostly done ... needs polish)
    • need to store action metadata in invoice for good retry breadcrumb
    • poll vote retries?
    • downzap retries?
  • [ ] freebies with attached wallets?
  • find solution to pending payment edits
    • opted to start the edit timer at invoicePaidAt
  • explicitly type all prisma queryRaw input types
    • prisma will inconsistently throw nondescriptive errors when input types are not what it expects
  • write docs for properly implementing paidActions
  • insane amounts of testing
  • big self-review

testing

Because this is such a behemoth, we need to test:

  1. parity with the code this replaces
  2. everything this intends to do
  3. concurrency is handled well, which is mostly verifying invoice state transitions only happen once, which is dependent on:
    • prisma throwing when record is not found on an update and transaction is in read committed isolation mode
    • postgres not doing dirty reads/writes within a single update statement
    • excessively reviewing our code to make sure there are no read-modify-write within paid action transactions

parity

This has the biggest impact on

  • item creation and updates
  • item acts, eg zaps and downzaps
  • the aggregate queries, eg stats, wot, rewards

... because we no longer do these in plpgsql, or for the aggregates, they need to filter for invoiceActionState. For all these, it'll help to enumerate everything that the old code does and make sure it still works.

intentions

  • all post types, comments, all item acts, all territory actions, donations, and updating all post types and comments support:
    • paying with fee credits
    • paying with qr code
    • paying with attached wallets
  • all post types, comments, zaps, downzaps, and poll votes support "optimism":
    • their data persists even on payment failure, but is only seen by the "actor"
    • payments can be retried ad infinitum
  • all post types, comments, zaps, and donations can be performed by @anon
  • all territory actions, post updates, comment updates can be done pessimistically but not by @anon
  • donations can be done pessimistically and also by @anon

testing TODOs

  • test optimistic and pessimistic payments with and without attached wallet
  • item creation
    • testing payments
      • fee credits
      • optimistic payments (logged in)
        • payment success
          • everyone can see it
            • ncomments
            • in item lists
            • fees in rewards
        • payment failure
          • no one can see it but the OP
          • can't be commented on
          • can't be zapped
          • receives notification indicator
          • show up in notifications
          • can be retried in place and in notifications
      • pessimistic payments (anon)
        • payment success
        • payment failure
    • link
      • just title and url
      • with context
        • delete and remindme bots
        • mentions
        • images
      • with boost
        • verify boost fee is taken and denormalized
      • with forwarding
        • verify zaps are split
      • deletion
    • comment
      • with delete and remindme bots
      • with images
      • without either
    • discussion
      • just title
      • title and text
    • poll
      • title and 2 options
      • title, text, and 2 options
      • title, text, and max options
    • bounty
      • just title
      • title and text
    • job
      • with the works without promotion
      • with promotion
  • item updates
    • link
      • context can be added/removed
      • boost can be increased
        • without fee credits is pessimistic
      • forwards can be added/removed
      • delete and remindme bots can be added/removed
      • images can be added/removed
    • discussion
      • text can be added/removed
    • poll
      • text can be added/removed
      • poll options can be added
    • bounty can be increased/decreased
    • job
      • promotion can be stopped/restarted
      • fields can be updated
      • can be deleted
  • zaps/downzaps
    • testing payments
      • can be done with fee credits
      • can be done optimistically
        • payment success
          • seen by everyone
          • increases weighted votes
        • payment failure
          • only seen by op
          • triggers notification indicator
          • seen in notifications and can be retried
      • pessimistically
        • success
        • failures
  • poll votes
    • testing payments
      • works with fee credits
      • done optimistically
        • payment success
          • seen by everyone
        • pending
          • only seen by op
        • payment failure
          • triggers notification indicator
          • seen in notifications and can be retried
  • territory creation and updates and billing payments
    • changing names/post types/post cost
    • unarchiving
    • paying with fee credits/qr code/attached wallet
    • auto-renew works with fee credits but not without
  • rewards/historical stats do not reflect pending/failed/held actions
  • wot does not reflect pending/failed/held actions

@huumn huumn changed the title wip backend optimism backend optimism May 27, 2024
@huumn huumn mentioned this pull request May 27, 2024
11 tasks
@huumn huumn changed the title backend optimism backend payment optimism May 28, 2024

This comment was marked as resolved.

lib/apollo.js Show resolved Hide resolved
lib/constants.js Show resolved Hide resolved
package.json Show resolved Hide resolved
worker/streak.js Show resolved Hide resolved
huumn added a commit that referenced this pull request Jun 29, 2024
Copy link
Member

@ekzyis ekzyis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a partial review. I read every single new line in the backend now and made sure I understand what it's doing. Left comments where I had questions.

Only components/, fragments/ and pages/ is left to review.

Looks good so far!

components/use-paid-mutation.js Outdated Show resolved Hide resolved
components/item-act.js Outdated Show resolved Hide resolved
api/resolvers/notifications.js Show resolved Hide resolved
components/notifications.js Show resolved Hide resolved
api/paidAction/README.md Show resolved Hide resolved
api/paidAction/lib/item.js Outdated Show resolved Hide resolved
api/paidAction/index.js Outdated Show resolved Hide resolved
api/resolvers/user.js Show resolved Hide resolved
Comment on lines 385 to 388
await models.$executeRaw`
INSERT INTO pgboss.job (name, data, retrylimit, retrybackoff, priority)
VALUES ('checkInvoice', jsonb_build_object('hash', ${hash}), 21, true, 100)`
return await models.invoice.findFirst({ where: { hash } })
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why not import checkInvoice and run it ourselves?

Like this, this means that the invoice isn't actually cancelled yet when the mutation returns.

Even the response would indicate that. So a hypothetical user interface where we have a cancel button next to the invoice state, the invoice state wouldn't update on cancel (since it's not cancelled yet).

We don't have this anywhere yet afaict but I considered being able to cancel deposits manually (instead of letting them expire) in the past.

api/resolvers/wallet.js Outdated Show resolved Hide resolved
@huumn
Copy link
Member Author

huumn commented Jun 30, 2024

Excellent review! You caught a lot of good stuff

Copy link
Member

@ekzyis ekzyis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Another review pass. Noticed a bug with edits. See comment.

components/invoice.js Outdated Show resolved Hide resolved
components/use-item-submit.js Outdated Show resolved Hide resolved
components/use-item-submit.js Show resolved Hide resolved
sub: item?.subName || sub?.name,
boost: boost ? Number(boost) : undefined,
bounty: bounty ? Number(bounty) : undefined,
maxBid: maxBid || Number(maxBid) === 0 ? Number(maxBid) : undefined,
Copy link
Member

@ekzyis ekzyis Jun 30, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not pointing out potential operator precedence confusion in less critical code, only where it's really important and thus warrants a double-check.

let { data, ...rest } = await mutate(innerOptions)

// use the most inner callbacks/options if they exist
const { onPaid, onPayError, forceWaitForPayment, persistOnNavigate } = { ...options, ...innerOptions }
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is forceWaitForPayment still used? I don't see it passed in anywhere.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, I don't think so. It's another solution to the problem persistOnNavigate solves.

}))
// block until the invoice to be marked as paid
// for pessimisitic actions, they won't show up on navigation until they are marked as paid
await invoiceWaiter.waitUntilPaid(invoice, inv => inv?.actionState === 'PAID')
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand the reason for this but these different definitions of "paid" depending on context are confusing. In the context of LND, "paid" means "LND settled the incoming payment" while in the context of actions, "paid" means "action completed successfully".

Maybe that's an indicator we shouldn't name the action state PAID but SUCCESS or something?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LND doesn't call it paid. It calls it settled and held. We call it paid and held.

If we zoom out, afaict the confusing thing is the name of this function here which suggests that paid has multiple meanings. I didn't want to rename this function initially, but I can rename it.

api/resolvers/item.js Outdated Show resolved Hide resolved
Copy link
Member

@ekzyis ekzyis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Final review

Requesting changes due to the lnd bug mentioned in the previous review and didn't want to override the review conclusion.

components/form.js Outdated Show resolved Hide resolved
Comment on lines 30 to 38
useEffect(() => {
if (!invoice) {
return
}
if (waitFor && waitFor(invoice)) {
onPayment?.(invoice)
}
setExpired(new Date(invoice.expiredAt) <= new Date())
}, [invoice, onPayment, setExpired])
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Related to #1195 (comment)

components/use-paid-mutation.js Show resolved Hide resolved
@huumn huumn merged commit ca11ac9 into master Jul 1, 2024
6 checks passed
@huumn huumn deleted the backend-optimism branch July 1, 2024 17:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants