Use scoped app storage on Android

[rubenwardy]

This must be merged and released with sufficient time before November 2021, to avoid data loss

From November 2021, the Play Store will no longer be accepting updates to apps which use the deprecated getExternalStorageDirectory() API.

Therefore, this commit replaces uses of the deprecated API with the new scoped API (getExternalFilesDir() and getExternalCacheDir()). It also provides a temporary migration to move user data from the shared external directory to new storage.

See https://developer.android.com/training/data-storage/ and https://developer.android.com/about/versions/11/privacy/storage

Fixes #2097, fixes #11417, fixes #11299. Potentially fixes #11462

To do

• Improve progress indication when migrating. Currently, nothing appears on screen. There should either be a progress bar or an indeterminate spinner on screen whilst the unzip service is doing something
• Check externalFilesDir state before launching app, to gracefully show an error
• Allow the UI to exit and recreate without requeing the service
• Check fault tolerance
• Testing

How to test

⚠️ Warning! Make sure to backup any important files before testing ⚠️
⚠️ This will move any worlds from the shared storage to the app storage. When the app is uninstalled, all stored data will be deleted - including worlds. Additionally, I can't guarantee it won't mess up ⚠️

APK builds: https://github.com/rubenwardy/minetest/releases/tag/5.5.0-scoped1

Perform the following on Android 11, 10, and 4/5:

• Check migration
  • Create worlds on the old version of the app
  • Upgrade
  • Run Minetest again, check that the worlds still appear
• Check new install
  • With no Minetest data, check that you can install the PR version and it works

[rubenwardy]

Builds: https://github.com/rubenwardy/minetest/releases/tag/5.5.0-scoped1

[SmallJoker]

For the logs: Tested on Android 6.0, armv6l. Crashes on startup, no error logs found so far.

[Warr1024]

* Check new install
  * With no Minetest data...

To confirm, I assume this means...

1. Uninstall any existing minetest app
2. Using a file manager, delete the shared-storage Minetest folder (in /sdcard or /storage/emulated/0 or whatever depending on Android version)

Is this correct?

[Warr1024]

Tested: Android 11 build RP1A.201005.004.A1 (stock from Google, unrooted) on Google Pixel 2.

Migration case: works as expected, though it took a long time with what I assumed were not large worlds. The screen went to sleep while I was waiting, and when I woke it up, MT was gone, but it appeared to have worked anyway (took me to main menu with settings and worlds apparently intact) though I did not thoroughly test all worlds.

Fresh install case: worked approximately as I expected. No games were installed by default, which is fine to me (debatable?), but upon trying to create a new world it told me "no games installed, download one from minetest.net" instead of directing me to the Content tab.

---

Further research:

• Does MT recover gracefully from a failed migration (screen locked, app killed, device restarted)?
• How long should migration normally take? Is this an issue for users on battery? Do users need to be warned? Have an option to quit instead and retry migration later?
• What are our plans for import/export? Is there a related issue already in GH?

[sfan5]

The screen went to sleep while I was waiting

Preventable by window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON) or just set android:keepScreenOn="true" on the activity

[rubenwardy]

Is this correct?

Yes

works as expected, though it took a long time with what I assumed were not large worlds.

Copying lots of small files with permission checks takes time, unfortunately :(

Copying games is what takes the most time currently. Perhaps it should exclude MTG?

The screen went to sleep while I was waiting, and when I woke it up, MT was gone

The sleep is probably desirable, but MT being gone is not. It should also correctly recover if you exit and return to the app. It uses a background service, so this should be possible

Preventable by window.addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON) or just set android:keepScreenOn="true" on the activity

I don't think this is desirable, for battery reasons. It's probably better to make it so you can close the app, and it continues in the service. And of course is fault tolerant

Fresh install case: worked approximately as I expected. No games were installed by default, which is fine to me (debatable?), but upon trying to create a new world it told me "no games installed, download one from minetest.net" instead of directing me to the Content tab.

I built the APK with Minetest Game installed, so it should have worked...

Does MT recover gracefully from a failed migration (screen locked, app killed, device restarted)?

It should do - the migration starts based on the existence of the Minetest folder in legacy storage, and the moving of files is done by Apache which should be safe. Worth testing

How long should migration normally take? Is this an issue for users on battery? Do users need to be warned?

It takes less than 30 seconds for me

Have an option to quit instead and retry migration later?

This could certainly be done, I could have a dialog each time you boot Minetest. Would add some complexity though

What are our plans for import/export? Is there a related issue already in GH?

Yeah, this will definitely be a problem. For export, I would like to get the Minetest C++ to zip up a world to the cache directory. The Java can then start a share intent, allowing users to save to the file system or share via an app or email. For import, I'd like to do the inverse - allow selecting a zip file, and extracting it

There's no issue for this yet

[Warr1024]

I built the APK with Minetest Game installed, so it should have worked...

Okay, it was my bad then, I used my own build, where I didn't know I needed to install MTG, since I assumed that I'd need to have both a before and an after build with the same signing key, in order to test the upgrade path.

That being said, MT does work alright (aside from the message text) when MTG is not installed, and on some level that's a better experience, though that's probably a topic for a separate issue. 😅

This could certainly be done, I could have a dialog each time you boot Minetest. Would add some complexity though

Mostly I'm thinking it might not be a bad idea to give users some kind of warning that it wouldn't be a bad idea for them to try to make their own manual backups independently before proceeding, just in case. It's a big ecosystem and we won't be able to test everything, and not all the affected users will be following engine dev scuttlebutt.

For import, I'd like to do the inverse - allow selecting a zip file, and extracting it

Making MT able to act as a share target for zip files, or as an associated app for zip files for file managers, could also be nice. It wouldn't be unprecedented to use our own file extension (e.g. *.mtzip) or something to make it unambiguous.

[rubenwardy]

If you close and reopen the app, Minetest will detect the running background service and reattach to it

[rubenwardy]

A progress bar is now shown in the notification

[sfan5]

Looks solid

[sfan5]

Due to time sensitivity we should release an Android-only 5.4.2 with this PR backported.

[rubenwardy]

Please note than Krock is experiencing issues on Android 10. This will need to be properly tested and debugged on older version, I only have access to Android 11

I can look at getting the app to compile under x86 so I can use it in an emulator

[snowyu]

No Problem on Android 10, But Crashed on Android 5.1.1(sdk22):
Linux localhost 3.0.72-DC-ga873bd8 #2 SMP PREEMPT Thu Apr 12 07:16:46 CEST 2018 armv7l GNU/Linux

E/AndroidRuntime( 9207): FATAL EXCEPTION: IntentService[net.minetest.minetest.UnzipService]
E/AndroidRuntime( 9207): Process: net.minetest.minetest, PID: 9207
E/AndroidRuntime( 9207): java.lang.NoClassDefFoundError: org.apache.commons.io.-$$Lambda$9BAXxIVvTQTfZLonlFIMrHz1T5w
E/AndroidRuntime( 9207): 	at org.apache.commons.io.IOUtils.<clinit>(IOUtils.java:183)
E/AndroidRuntime( 9207): 	at org.apache.commons.io.IOUtils.copy(IOUtils.java:953)
E/AndroidRuntime( 9207): 	at org.apache.commons.io.FileUtils.copyToFile(FileUtils.java:1043)
E/AndroidRuntime( 9207): 	at net.minetest.minetest.UnzipService.onHandleIntent(UnzipService.java:85)
E/AndroidRuntime( 9207): 	at android.app.IntentService$ServiceHandler.handleMessage(IntentService.java:65)
E/AndroidRuntime( 9207): 	at android.os.Handler.dispatchMessage(Handler.java:102)
E/AndroidRuntime( 9207): 	at android.os.Looper.loop(Looper.java:135)
E/AndroidRuntime( 9207): 	at android.os.HandlerThread.run(HandlerThread.java:61)

[Emojigit]

load permanently on android 5 and no process bar

[snowyu]

It can worked after removing org.apache.commons.io.FileUtils on Android 5.1.1.

// import org.apache.commons.io.FileUtils;

	@Override
	protected void onHandleIntent(Intent intent) {
....
			try (InputStream in = this.getAssets().open(zipFile.getName())) {
				copyToFile(in, zipFile);
				// FileUtils.copyToFile(in, zipFile);
			}

			// migrate(notificationBuilder, userDataDirectory);
			unzip(notificationBuilder, zipFile, userDataDirectory);
....

	}

	private void copyToFile(InputStream src, File dest) throws IOException {
		try (OutputStream outputStream = new FileOutputStream(dest)) {
			int readLen;
			byte[] readBuffer = new byte[8192];
			while ((readLen = src.read(readBuffer)) != -1) {
				outputStream.write(readBuffer, 0, readLen);
			}
		}
	}

/* comments migrate
	private void migrate(Notification.Builder notificationBuilder, File newLocation) throws IOException {
        // ....
	}
*/

[sfan5]

Here's what could work to avoid importing commons-io:

    boolean moveDirectoryToDirectory(@NonNull File src, @NonNull File dst) {
        try {
            Process p = new ProcessBuilder("/system/bin/mv",
                    src.getAbsolutePath(), dst.getAbsolutePath()).start();
            return p.waitFor() == 0;
        } catch (IOException | InterruptedException e) {
            return false;
        }
    }

[sfan5]

Removed problematic library.
Tested:

• Migration Android 11 (takes at least 5 minutes)
• Fresh install Android 11
• Migration Android 7 (pretty much instant)
• Fresh install Android 7

[sfan5]

Test APK: 550-with-migration-fixed.zip

@rubenwardy please review

[rubenwardy]

Tested on Android 11. The changes you made look good to me, thanks for picking this up

[snowyu]

• Migration on Android 5.1.1 OK
• Fresh install on Android 5.1.1(8 minutes) OK