localStorage wiped on 0.12.3 upgrade and subsequent app launches

[matthew-dean]

I've just started testing my app with 0.14.5 instead of 0.12.3. On first launch, all my app's settings in localStorage were wiped. (Similar to #4527, except not using an app:https:// protocol. Just referencing localStorage). I thought maybe this was a migration bug, but it also seems that localStorage is wiped for each app launch. Are there any oddities with localStorage that could explain this, or any known workarounds?

[matthew-dean]

With more testing, I'm finding that eventually I could get localStorage written, but it's inconsistent. It sometimes reads, sometimes writes when the app is loading. I'm not sure the pattern of when it's succeeding. I notice in the local storage setting in devtools, it lists my app's data under chrome-extension:https://. Not sure if that indicates some change.

[mscreenie]

I think this is related or similar to the following issues:
#4747
#4856
#4757
#4469

Are you finding the localStoage is completely gone in 12.3 after you open 14, 15? In my case it still exists. It just seems 12.3 and 14+ are using separate databases. I cannot seamlessly upgrade my users to newer versions. It would involve an ugly fix perhaps platform specific fix.

[matthew-dean]

@mscreenie I don't know that it's gone as in not present on the local system. But the data definitely isn't there in app. And... saving localStorage seems buggy as hell. Like... a few times launching the app it didn't save settings. I'm not sure of the reason. It eventually seemed to persist the data, so that's baffling too.

It looks like the solution posted on one of those issues might be the best: migrate localStorage to a different kind of data storage solution in the 12.3 version, and then use that data storage solution in 14/15. Buuuut not everyone is going to upgrade, so that's still not perfect. If I had known localStorage would eventually be broken I wouldn't have used it to begin with, but it seemed like a reliable native solution.

So, probably I'll migrate from localStorage, and then stay on 12.3 and hope and pray that eventually the localStorage issue is fixed for people who don't upgrade before I would switch to something after 12.3.

[orther]

My app is also not migrating the data properly from v0.12.3 to v0.14.5. What I noticed is that I can copy my localStorage and IndexedDB files manually and the data is available.

I assume this code is supposed to do this automatically but is failing to do so for some reason: https://github.com/nwjs/nw.js/blob/nw14/src/nw_content.cc#L332

I can't yet determine why the data isn't being automatically copied over on the first run of the v0.14.5 app like expected. It should be noted that I am experiencing this on Windows.
UPDATE: I can't reproduce the localStorage migration failure on OSX but I thought I could yesterday.

NOTE: To manually copy the localStorage files and IndexedDB folders from the v0.12.3 app's data dir (window.nwDispatcher.nwGui.App.dataPath) to the v0.14.5 app's data dir (nw.App.dataPath) I had to rename the files/folders from file__0.* to the new chrome-extension_myappextensionidgoeshere_0.* naming scheme.

[orther]

I am having success with my initial testing of duplicating the migrate data functionality of https://github.com/nwjs/nw.js/blob/nw14/src/nw_content.cc#L332 into JS that is ran at the beginning of my app boot. I am getting crashes after the JS version of migration (which only happens if localStorage and/or IndexedDB data exists) but on the next boot the user data is available and the app works as usual. I'm hoping manually triggering a nw.App.reload after my JS migration solves this. If I can get this code working consistently I will try to post it here.

[matthew-dean]

What I've been able to do so far is circumvent localStorage / IndexedDB entirely by using a library that efficiently persists an in-memory object to disk. Which means I'll probably be shipping my app with 0.12.3 for a long while so that the migration script can run on user's copies of the app. In my initial testing, the file works fine in 0.14.5 since the app/user data location is the same.

Obviously, I would prefer if this were fixed, but seeing as how this was overlooked in several versions of NW.js, I don't think the "in browser-context" storage solutions can be trusted. Probably the browser should be thought of as a VM that may not persist. In the web environment, it's known that any storage is, at best, temporary. For this reason, I felt a little queasy about using localStorage in the first place, since it was designed for something that's not a permanent app environment.

STILL, fixing this would be great.

[matthew-dean]

This is what I'm migrating my localStorage objects to: https://github.com/mafintosh/flat-file-db

One of the reasons I liked it is that it has a synchronous option for connecting the DB, and gets are synchronous, which makes it an easy drop in for localStorage. Puts are async, but that's even better, since it doesn't block the thread.

[orther]

Yeah we're testing this JS based migration solution thoroughly and if we can get it working our plan is to migrate to 0.14.5 but then move to a non-native data storage solution. The one we're eyeing currently is: https://github.com/louischatriot/nedb

[matthew-dean]

@orther I looked at that too. It looks pretty cool. In my case, my "database" requirements were very small, as it was basically "user preferences" I needed to store, so a file-based store was better.

[mscreenie]

I've eyed moving to a js based dB however backward comp is a basic feature that should be worked on before a new version comes out of alpha or rc. This being a known issue since 13, Would seem this is has been ignored again in the latest release. Replacing a native function with a non native function doesn't seem to be a good solution, although I can see why it's appealing if a developer is desperate.

[rogerwang]

To all people who hit localstorage migration issue, could you please submit a sample app with the user data dir? We do support the data migration and have a test case to make sure it doesn't break: https://github.com/nwjs/nw.js/tree/nw14/test/sanity/migrate-localstorage

[matthew-dean]

@rogerwang How is the data migrated? Like I noted at the start, the behavior was not consistent. I seem to remember seeing my local storage wiped again on 0.14.5. In other words, localstorage from 0.14.5 to 0.14.5 was not persisted. But then I couldn't duplicate the behavior later, so something else may have gone wrong in that instance.

If the data is literally "migrated", that wouldn't be a solution for me either, because then that would suggest that localstorage would not persist when downgrading.

[orther]

I have a theory that this is NOT an issue with 0.14.5 migrating data from old versions but instead a result of running the old 0.12.3 nw.js version of the app AFTER running the new 0.14.5 version. Since the user data files have been moved to the new location after running the 0.14.5 version, the 0.12.3 version will start and create new empty localStorage/IndexedDB files. If the 0.14.5 version is ran after that the newly created empty localStorage/IndexedDB files from the 0.12.3 version are copied OVER the existing 0.14.5 versions. This results in empty localStorage/IndexedDB.

After testing this theory a few times I believe it to be the cause. This would explain why the behavior seemed inconsistent. If this is the case, a user can wipe their data by running the old version of the app which is scary.

[rogerwang]

@matthew-dean the 0.12 data is moved to the new location after 0.13.

@orther good point. I'll fix this case by skip the moving if 0.13 data file is there.

[matthew-dean]

@orther Which reflects my concern. And, indeed, I do end up running 2 versions concurrently. Both need to work with the same data set. It seems like the clear solution is to not move the data at all, and use a different approach in > 0.13 to access the data that is already present, but maybe that's technically difficult.

[orther]

@rogerwang do you think a v0.14.x maintenance version bump with this fix will be released anytime soon?

[orther]

Also I'm curious why data migration was necessary in the first place. Is that a result of changing to chrome-extension:https:// protocol?

[reznikartem]

My app lost localStorage too, after upgrade from 0.12.2 to 0.15.0 :(

[rogerwang]

@orther it's already released in 0.14.6 and 0.15.1.

[rogerwang]

@orther it's changed due to protocol, as well as the architecture optimization we've done in 0.13.

[havardholvik]

This is still an issue for me as well @rogerwang. I've made a quick video to show how I can reproduce this; https://www.youtube.com/watch?v=IX6VFNO4oro&feature=youtu.be . In our real app indexedDB also fails to migrate as well. As of now, this is the only issue preventing us from upgrading to nwjs-0.14/0.15.

This was shot on Windows 7 SP1.

[orther]

Yes I can replicate this issue following the techniques in @havardholvik video. I tested with 0.12.3 to both 0.14.5sdk and 0.14.6sdk versions. The localStorage does not copy over.

[rogerwang]

To all people who have issues: please upload a sample with user data dir of 0.12 so I can reproduce it.