🚀 Feature: Bulk Document Creation

[Shadowfita]

🔖 Feature description

Create a "createDocuments" post endpoint that takes an array of documents.

🎤 Pitch

In my project, I am trying to insert 12,000 documents in one go. It is inefficient running 12,000 external createDocument API calls to achieve this.

To remedy this, there should be a "createDocuments" endpoint that allows you to pass an array of documents. That way you could easily break up the number of external calls required, and allow the stack to process the creation of the large amount of documents interally, quickly and efficiently.

A potential workaround would be creating a function that does it locally, but I don't believe this is a proper solution.

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[Shadowfita]

This is not achieveable with functions due to the 8192 character limit.

[Shadowfita]

I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.

[Shadowfita]

I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.

It's currently taking about 46 seconds to process 15,863 simple json objects.

[zcoderr]

Need this feature too

[elunatix]

I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.

Hi, can you point me to a demo file please. NodeJS if possible.

[Shadowfita]

I have a achieved an okay workaround by creating a ".json" file inside a bucket that contains an array of documents, which is then read by a function that inserts the required documents.

Hi, can you point me to a demo file please. NodeJS if possible.

No worries.

You can find it at the link below. It was thrown together pretty quickly.

It expects a JSON file to be created in a bucket with the following structure

{ 'collection': 'collection_id', 'data': [ 'object-array' ] }

https://gist.github.com/Shadowfita/b5ccd20f65566cb9f2b40d416c5201a2

[pilcrowOnPaper]

Would also like a batch delete too.

[FuadEfendi]

Using functions and bucket is brilliant solution!

Another potential workaround is to use multithreaded client and upload documents in parallel; for instance, client application developed in Kotlin.
And, perhaps Server SDK can allow to create server-side extension; or, trivial, server-side application (such as "function", or Kotlin-based standalone) can read batch JSON from local filesystem (from "bucket", FTP, S3, etc) and create records, so we avoid excessive HTTP traffic.

Or, another solution:

• use "Staging" instance in local network, load data, backup MariaDB, restore MariaDB in production system
• backup production MariaDB, analyze SQL dump file, programmatically generate additional 12,000 SQL statements, and restore it

The best approach would be if Appwrite API has such functions: export (permissions/objects/buckets/collections/functions) (JSON), import, etc.; in this case we "abstract" underlaying implementation details and can do more granular export/import. For example, right now we don't have implementation-independent backup/restore (except that executing sqldump locally; what about cluster then?)

[balachandarlinks]

It would be great if bulk update is also considered as it ll greatly reduce the number of requests I make in my application.

[singhbhaskar]

Hey @stnguyen90 ,
Can I work on this. I have gone through the requirements and tried to implement a basic endpoint to achieve this and was able to implement this on my local setup for Bulk create. I would be happy to contribute to this.

[stnguyen90]

@singhbhaskar, thanks for your interest! 🙏 However, it would be best for the core team to figure out how it should work.

[rafagazani]

Need this feature too

[danilo73r]

I Need bulk operations too, please

[Vedsaga]

I need this also, need to create 18K document which are Pincode I should say...

[ashuvssut]

I want to delete multiple docs by their IDs too

[Shiba-Kar]

So I am working on a scrapping project , there are almost 2K records , it takes almost 20 min to write !!!

{"ColumnRef":"Name","index":"0","value":"orange house"}

this is the size of each record !!!

is there any way to speed up the writes !!

[Shadowfita]

So I am working on a scrapping project , there are almost 2K records , it takes almost 20 min to write !!!

{"ColumnRef":"Name","index":"0","value":"orange house"}

this is the size of each record !!!

is there any way to speed up the writes !!

You should run all write requests asynchronously and wrap them in Promise.all . Appwrite is built to scale and will handle that many concurrent requests with no issue, and it will reduce your wait-time exponentionally.

[ashuvssut]

@Shadowfita Is there a guarantee that all promises will resolve if we do it with Promise.all now?

Last time I checked like 2 months ago, I tried to do bulk document deletion with Promise.all and it seems some of them failed.

[Vedsaga]

Is there a guarantee that all promises will resolve if we do it with Promise.all now?

Answer depends actually,

INstead you should delete in batches, meaning 10-50 per Promise instead of all at one. if you do all at one then it will likely fail because of limitation of the server.

[tripolskypetr]

Right now I am using that lib for bulk document-from-json creation

[jquiros2]

I agree a csv importer in the dash would be very useful.

I also had gone down the road of: csv--> json, then json--> appwrite but throttling and processing just made it unbearable for larger workloads. This is using appwrite self-hosted.
you need to figure out what your own _3_database_x_collection_x table is.

use appwrite;
LOAD DATA LOCAL INFILE '/path_to_your.csv'
INTO TABLE _3_database_x_collection_x
CHARACTER SET latin1 
FIELDS TERMINATED BY ','
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(list, the, fields, in, your csv)
SET _uid = LEFT(UUID(), 20), 
_createdAt = DATE_FORMAT(NOW(), '%Y-%m-%d %H:%i:%s.%f'),
_updatedAt = DATE_FORMAT(NOW(), '%Y-%m-%d %H:%i:%s.%f'),
_permissions="[]";

For _uid I had used LEFT(MD5(RAND()), 20) at first but I got too many repeated values.

For testing, best to copy the _3_database_x_collection_x table into another and feed that one. then

insert into a select * from b

This seems to work. ymmv and proceed at your own risk.

[TechComet]

Any new ?

[tripolskypetr]

Guys use https://github.com/react-declarative/appwrite-backup-tool

The restore script can upload over than 10_000 documents without loading all of them to the RAM. That means it scalable and work with any size of data