🐛 Bug Report: very slow migration

[AidenY69]

👟 Reproduction steps

When upgrading my current AppWrite to the newest version, the migration takes multiple days.

👍 Expected behavior

Faster migration regardless of record size

👎 Actual Behavior

When upgrading my current AppWrite to the newest version, the migration takes multiple days. I have ~30 million records, but I feel like that shouldn't be impacting the migration since the database itself isn't changing, just the AppWrite code. Is there any solutions on how I can make this process faster, or is there something on AppWrite's end that you can do to optimize large migrations?

The platform has to be down during a migration, and with a 4 day estimation, that's pretty extreme for a scheduled downtime.

🎲 Appwrite version

Version 1.3.x

💻 Operating system

Linux

🧱 Your Environment

No response

👀 Have you spent some time to check if this issue has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

[joeyouss]

Hi!
Thank you for bringing this up, yes we are aware of this happening and we have a WIP PR for this #5876

[stnguyen90]

@AidenY69, what versions are you upgrading between?

I feel like that shouldn't be impacting the migration since the database itself isn't changing, just the AppWrite code

When running migrations, the database schema and documents are updated and Appwrite must iterate over everything.

[tripolskypetr]

Right now I am using these PowerShell scripts because I need to export data into JSON format. That method is simple and guarantee the data to be readable, combinable from deferent sources and storable in Git
https://github.com/react-declarative/appwrite-backup-tool

[tripolskypetr]

@joeyouss I think JSON format export feature will be quite useful for AppWrite CLI

for (const collection of collections) {
  console.log(`Making backup of ${collection.$id}`);
  await fs.mkdir(`backup/databases/${database.$id}/${collection.$id}`, {
    recursive: true,
  });
  for await (const document of listDocuments(database.$id, collection.$id)) {
    console.log(
      `Writing ${document.$id} from ${collection.$id} from ${database.$id}`
    );
    await fs.writeFile(
      `backup/databases/${database.$id}/${collection.$id}/${document.$id}.json`,
      JSON.stringify(document, null, 2)
    );
  }
}