🚀 Feature: Immutable storage (blockchain/IPFS) for censorship resistance

[TechupBusiness]

🔖 Feature description

Ability to store data on an immutable storage (IPFS), gun.js or a cheap but good enough blockchain/side-chain (or even several chains) instead of only locally. Ethereum is for sure a good option but the transaction costs are way too high. But the EVM (Ethereum virtual machine) is also used on many other blockchains, so being compatible with EVM blockchains would already allow the use of many different blockchains, which increases the censorship resistance.

The commercial cloud fork of parse back4app is working already on such an adapter, but only supporting the super expensive main chain which makes it unusable: https://www.back4app.com/docs/get-started/blockchain-data-storage
And its not clear yet if it's the right approach.

Another potentially promising database layer for appwrite could be IPDB (Interplanetary Database, similar to IPFS but not just a file storage), using: BigChainDB (or using it directly, but this would require spanning up an own blockchain network) https://www.bigchaindb.com/

I'm welcoming discussions on this topic, what would be the easiest approach to implement such features best into appwrite without too many compromises. And find the best solutions for choosing the immutable storages.

🎤 Pitch

As censorship resistance and resilience becomes more and more important these days (bank accounts frozen, sites shut down, fact checkers decides whats true even if a few month later the truth was wrong, witch hunts (assange, snowden, etc.), countries bombing other countries and shutting their IT infrastructure down) it would be awesome to have an open source platform that can act as perfect easy-to-use layer for decentralized autonomous organizations (DAOs) and other free speech / knowledge applications. More and more countries adopt the idea of absolute surveillance (China as a template) and its becoming increasingly more difficult and dangerous to run or create neutral platforms. Blockchain and immutable storages (IPFS) were born out of this issue of the lack of free speech and monetary control of central authorities that dont always act in the best of the people. Unfortunately there is NO SOFTWARE in existance at the moment that gives developers the ability to write easily apps for neutral software with privacy in mind and without a single attack vector (one server of the initiator). But there are already decentralized container clouds available that could run appwrite, like Flux.

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[tessamero]

@TechupBusiness This is something that has been on the top of my mind (and haven't really discussed out loud yet) for Appwrite Cloud.

I think it would be good for this to be an open discussion for different options and for people to share their use cases of how/why they would need it (in generic terms) so there is more prioritization on implementing blockchain.

Some tools I was looking into were https://chain.link/ and also https://nft.storage/

What are your thoughts on moving this into a discussion? It would be further down in the roadmap if this was something we added, especially with the amount of web3 enthusiasts we have at the company.

@christyjacob4 take a look at this =)

[christyjacob4]

@tessamero @TechupBusiness this looks really cool. Currently, all of Appwrite's external storage providers ( DigitalOcean Spaces, Amazon S3, Linode, Backblaze, MinIO etc. ) have an S3 compatible API. If any of these storage solutions have an S3 compatible API, they can be supported right away. if not, we need some more work to get them to work with Appwrite.

But this is definitely something we'll work on adding support for 😄

[badmark]

@tessamero @TechupBusiness this looks really cool. Currently, all of Appwrite's external storage providers ( DigitalOcean Spaces, Amazon S3, Linode, Backblaze, MinIO etc. )

Is there a storage adapter for Minio? I cannot find a single example for the .env settings.

[stnguyen90]

Is there a storage adapter for Minio? I cannot find a single example for the .env settings.

Coming soon: utopia-php/storage#51

[badmark]

Very cool!

[TechupBusiness]

Thanks for your replies!!

@tessamero @TechupBusiness this looks really cool. Currently, all of Appwrite's external storage providers ( DigitalOcean Spaces, Amazon S3, Linode, Backblaze, MinIO etc. ) have an S3 compatible API. If any of these storage solutions have an S3 compatible API, they can be supported right away. if not, we need some more work to get them to work with Appwrite.

But this is definitely something we'll work on adding support for 😄

There are only a very few projects that have S3 support like https://filebase.com/ (commercial) and Storj Blockchain (https://www.storj.io/benefits/s3-compatible). But it would be for sure a good start if they work with the S3 adapter!

There are plenty other blockchains, focussing on storage or more like: Filecoin, Storj (S3 support as mentioned), Siacoin, Arweave or MaidSafe. Some have a "permanent" storage, others seems to support deletions.

Or projects that also wanna be more than just a storage like Internet Computer (https://dfinity.org/).

@TechupBusiness This is something that has been on the top of my mind (and haven't really discussed out loud yet) for Appwrite Cloud.

I think it would be good for this to be an open discussion for different options and for people to share their use cases of how/why they would need it (in generic terms) so there is more prioritization on implementing blockchain.

Some tools I was looking into were https://chain.link/ and also https://nft.storage/

What are your thoughts on moving this into a discussion? It would be further down in the roadmap if this was something we added, especially with the amount of web3 enthusiasts we have at the company.

@christyjacob4 take a look at this =)

Chainlink and NFT Storage are probably not that ideal for this case. Chainlink is an Oracle = Data provider and AFAIK not usable for storage of structured data for single projects. Technically it may be possible but its not the intention of the project to support storage of normal data (if I'm right). While NFT Storage seems to focus on NFTs. But its probably technically possible to also use it for any kind of data... ("like metadata, images, and other assets"). So definitely also an option.

I could do some more research first about the existing storage solutions, which would make sense to connect (apart from S3 compatible ones).

[tessamero]

@TechupBusiness Ahh yes, Filebase. I would actually prefer that over Arweave in terms of scalability/long-term efficiency. I've been following quite a few companies (and these) for a while. I can reach out to them and at least get a conversation started. :)

[TechupBusiness]

@TechupBusiness Ahh yes, Filebase. I would actually prefer that over Arweave in terms of scalability/long-term efficiency. I've been following quite a few companies (and these) for a while. I can reach out to them and at least get a conversation started. :)

Filebase is for sure a good option and already compatible with appwrite, if I understand it right? But the downside with such a service is, that its also just another single point of failure. They can be forced to stop offering their API to your project, which makes it unusable even if data is stored on these blockchains. Only a direct blockchain/immutable storage access or an open-source S3 API that everyone can run to access the on-chain data, would solve this issue (the app could make the s3 address configurable, so as a user you can simply connect to another S3 API in case your current API goes down or blocks the access). But I dont think its possible with the authentication/authorization model of S3 APIs.

Also Storj's "native" S3 adapter seem to have this "design flaw" (from a resilience perspective): https://www.storj.io/blog/what-is-s3-compatibility

[zdanl]

Feature description

Ability to store data on an immutable storage (IPFS), gun.js or a cheap but good enough blockchain/side-chain (or even several chains) instead of only locally. Ethereum is for sure a good option but the transaction costs are way too high. But the EVM (Ethereum virtual machine) is also used on many other blockchains, so being compatible with EVM blockchains would already allow the use of many different blockchains, which increases the censorship resistance.

The commercial cloud fork of parse back4app is working already on such an adapter, but only supporting the super expensive main chain which makes it unusable: https://www.back4app.com/docs/get-started/blockchain-data-storage And its not clear yet if it's the right approach.

Another potentially promising database layer for appwrite could be IPDB (Interplanetary Database, similar to IPFS but not just a file storage), using: BigChainDB (or using it directly, but this would require spanning up an own blockchain network) https://www.bigchaindb.com/

I'm welcoming discussions on this topic, what would be the easiest approach to implement such features best into appwrite without too many compromises. And find the best solutions for choosing the immutable storages.

Pitch

As censorship resistance and resilience becomes more and more important these days (bank accounts frozen, sites shut down, fact checkers decides whats true even if a few month later the truth was wrong, witch hunts (assange, snowden, etc.), countries bombing other countries and shutting their IT infrastructure down) it would be awesome to have an open source platform that can act as perfect easy-to-use layer for decentralized autonomous organizations (DAOs) and other free speech / knowledge applications. More and more countries adopt the idea of absolute surveillance (China as a template) and its becoming increasingly more difficult and dangerous to run or create neutral platforms. Blockchain and immutable storages (IPFS) were born out of this issue of the lack of free speech and monetary control of central authorities that dont always act in the best of the people. Unfortunately there is NO SOFTWARE in existance at the moment that gives developers the ability to write easily apps for neutral software with privacy in mind and without a single attack vector (one server of the initiator). But there are already decentralized container clouds available that could run appwrite, like Flux.

Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

Have you read the Code of Conduct?

• I have read the Code of Conduct

While your suggestion and thoughts are most valueable, I would like to submit the following quote with real world benchmarks and Metrics, as comparing a Blockchain or Appchain, Smartchain, with Serverless Functions, which one could turn Appwrite into, to a (FPGA driven) Redis Cluster with Isolated Execution Environments for NodeJS 19 and Typescript compiled to WebAssembly Ahead-of-time potentially outbidding JIT of v8 in NodeJS or your Browser.

@eldadfux

Despite the strength of blockchain, scalability issues (low throughput, high latency, storage issue and poor read-performance) are its big challenges but less researched [11]. Blockchain applications have much lower throughput compared to non-blockchain applications. For example, Bitcoin and Ethereum support 3–4 and 15–20 transactions per second (TPS) respectively compared to Visa and PayPal supporting 1667 and 193 TPS respectively [12]. On the other hand, query response (read performance) of blockchain servers is also poor compared to that of non-blockchain servers. For example, the query latency is over 10 ms in our blockchain system handling about 96 requests per second. Similarly, the Blockcypher blockchain server supports 3 requests per second [13]. In comparison with non-blockchain servers, Google and YouTube handle 84,405 requests per second and 85,021 views per second respectively [14], [15].

• https://www.sciencedirect.com/science/article/abs/pii/S0140366421000384