Add copy_if_not_exists support for AmazonS3 via DynamoDB Lock Support

[tustvold]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

#4239 added copy_if_not_exists support for S3-compatible stores that support it, i.e. everything other than S3 😅.

Having native support for this, and potentially other conditional operations in the future (#4879), would reduce user friction and I think unlock a number of exciting use-cases by making support consistent across the backends.

Describe the solution you'd like

I would like to be able to configure a DynamoDB to use in a manner compatible with the AWS DynamoDB Lock Client.

Describe alternatives you've considered

These is a Rust port of the AWS lock client, but it depends on rusoto, and doesn't appear to have very strong fencing guarantees.

Additional context

FYI @wjones127

[wjones127]

FWIW, the DynamoDB lock protocol hasn't become the preferred protocol for Delta Lake or Lance, mostly because of the dangers of lease-based locks. Spark Delta Lake went with a slightly different protocol, described in this doc. In Lance we did something similar. delta-rs still uses dynamodb-lock, but will eventually move to match the protocol used by Spark Delta Lake.

That being said, offering something decent out-of-the-box for S3 might be nice.

[tustvold]

Hmm... Yeah, I spent some time bashing at this, and it isn't easy to provide strong guarantees with this approach 😅

As iceberg also is doing something similar using DynamoDB to store the latest metadata file, I think lets not do this for now.