Cannot sync: runtime error - invalid memory address or nil pointer dereference

[jeroenmimpen]

My OS: Ubuntu 22.04.2 LTS

When running the most recent binary, I cannot sync. I had the same problem on the previous 2 binaries. Am I doing something wrong? If not: I have a large amount of projects and tasks in my todoist. Could this be a problem?

I ran the following commands:

$ wget https://github.com/sachaos/todoist/releases/download/v0.19.0/todoist_linux_amd64
$ mv todoist_linux_amd64 todoist
$ chmod +x todoist 
$ ./todoist sync

Resulted in:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x6b6211]

goroutine 1 [running]:
github.com/sachaos/todoist/lib.addToChildItem(...)
	/Users/tsakao/dev/src/github.com/sachaos/todoist/lib/sync.go:99
github.com/sachaos/todoist/lib.(*Store).ConstructItemTree(0xc000129c00)
	/Users/tsakao/dev/src/github.com/sachaos/todoist/lib/sync.go:176 +0x2f1
github.com/sachaos/todoist/lib.(*Client).Sync(0xc00013af00, {0x930d28, 0xc000126000})
	/Users/tsakao/dev/src/github.com/sachaos/todoist/lib/todoist.go:116 +0x1df
main.Sync(0xc0001731e0?)
	/Users/tsakao/dev/src/github.com/sachaos/todoist/sync.go:12 +0x6a
github.com/urfave/cli/v2.(*Command).Run(0xc0001731e0, 0xc00013af40, {0xc0001972c0, 0x1, 0x1})
	/Users/tsakao/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274 +0xa42
github.com/urfave/cli/v2.(*Command).Run(0xc000173600, 0xc00013a880, {0xc00013c000, 0x2, 0x2})
	/Users/tsakao/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:267 +0xc8a
github.com/urfave/cli/v2.(*App).RunContext(0xc0001fe000, {0x930d28?, 0xc000126000}, {0xc00013c000, 0x2, 0x2})
	/Users/tsakao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332 +0x605
github.com/urfave/cli/v2.(*App).Run(...)
	/Users/tsakao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
main.main()
	/Users/tsakao/dev/src/github.com/sachaos/todoist/main.go:345 +0x1f0d

I get the same error when cloning the repository and building it myself:

todoist sync
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x60 pc=0x6ba5d1]

goroutine 1 [running]:
github.com/sachaos/todoist/lib.addToChildItem(...)
	/home/user/go/src/github.com/sachaos/todoist/lib/sync.go:99
github.com/sachaos/todoist/lib.(*Store).ConstructItemTree(0xc0000d5c00)
	/home/user/go/src/github.com/sachaos/todoist/lib/sync.go:176 +0x2f1
github.com/sachaos/todoist/lib.(*Client).Sync(0xc0000baf00, {0x936440, 0xc0000240c0})
	/home/user/go/src/github.com/sachaos/todoist/lib/todoist.go:116 +0x1df
main.Sync(0xc000193340?)
	/home/user/go/src/github.com/sachaos/todoist/sync.go:12 +0x6a
github.com/urfave/cli/v2.(*Command).Run(0xc000193340, 0xc0000baf40, {0xc00021f2f0, 0x1, 0x1})
	/home/user/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:274 +0xa42
github.com/urfave/cli/v2.(*Command).Run(0xc000193760, 0xc0000ba880, {0xc000020040, 0x2, 0x2})
	/home/user/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:267 +0xc8a
github.com/urfave/cli/v2.(*App).RunContext(0xc0000001e0, {0x936440?, 0xc0000240c0}, {0xc000020040, 0x2, 0x2})
	/home/user/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:332 +0x605
github.com/urfave/cli/v2.(*App).Run(...)
	/home/user/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:309
main.main()

[HacDan]

I'll spin up a fresh 22.04 today to test this.

Can you try deleting your '/home/username/.cache/todoist/'directory to see if this makes a difference? (No Tilda on the Apple Keyboard apparently)

[jeroenmimpen]

Thanks for the response!

Can you try deleting your '/home/username/.cache/todoist/'directory to see if this makes a difference? (No Tilda on the Apple Keyboard apparently)

I just tried this; it had no effect. I still got the same error.

As a sidenote: I did not need to re-enter my API-token after deleting this cache, I assume that is stored elsewhere (just asking to make sure I removed the proper directory and it had the proper effect)?

[kenliu]

The API token is stored in ~/.config/todoist. AFAIK that is the only other place that Todoist reads anything from the filesystem.

Do you know roughly how many projects and tasks you have in your account? I have ~100 projects with ~1100 tasks for comparison.

[jeroenmimpen]

Do you know roughly how many projects and tasks you have in your account? I have ~100 projects with ~1100 tasks for comparison.

Roughly 3700 tasks, 170 projects

[kenliu]

Looking at this stack trace more closely, the line in question seems to be dereferencing a nil pointer. I'm not super familiar with this code, but what seems to be going on is that the parent variable is set to nil after calling FindItem, which is unexpected.

My guess is that this is a data issue that the todoist (cli) code is not handling properly and not an issue with your OS or configuration or number of projects/tasks. But without your actual todoist data, it's hard to figure out exactly what is causing this. Would you be open to helping troubleshoot this locally on your machine? Since you already have a go dev environment set up you could add some fmt.println statements in sync.go to identify which item is causing this to blow up and maybe look at the response from the todoist API to see if there is anything irregular with that particular item in the raw data from the server.

[HacDan]

I only have a setup with ~20 projects and ~1000 tasks, so I may not be able to reproduce this, but that shouldn't be exceeding any memory limits.

I just pulled the latest binary from GitHub on a fresh install of 22.04.2 and it syncs without issue. I followed the steps outlined. I will note this is a server (read: no GUI) install of Ubuntu 22.04.

Before adding some Println statements, could you try running with the --debug flag? This would need to be placed before any of the other operations, for example ./todoist --debug sync

I'm guessing it'll crash before it gets to any debug statements, but it's worth a shot.

[jeroenmimpen]

Would you be open to helping troubleshoot this locally on your machine?

Sure, never programmed in Go before, but I gave it a try. It seems to me like its the child of a completed task. The parent task is completed, but one of its children isn't. And it also isn't counted, since it says 7/7, but I count 8 myself. I guess my todoist data itself is broken, resulting in a situation that should be impossible.

I'm guessing it'll crash before it gets to any debug statements, but it's worth a shot.

Maybe no longer relevant, but I did it anyway: I did get some debug info with a request (containing probably sensitive cookie info, so I did not screenshot it) and a response. Afterwards the crash.

[HacDan]

Good point, I forgot it dumps the API calls, which include your sync key. My mistake there. That should probably be fixed, haha!

I'll see if I can reproduce your data set here to see if I can at least reproduce the crash. I don't think that's an impossible scenario.

Unrelated, I'm really curious what you were making with spinach and smoked chicken, haha!

[kenliu]

Thanks for troubleshooting! This is a pretty plausible theory. Fairly certain that this situation would cause a crash.

I think the official Todoist clients automatically mark all child tasks as completed, but that might not be enforced by the API and if you are using the todoist cli to update projects then your task data might wind up in this bad state. Would be interesting if you are able to repro this @HacDan.

[HacDan]

I can say for pretty much certain you cannot reproduce this exact scenario with the web or desktop client (under normal network conditions). One thing I missed at an initial glance was the 7/7 completed subtasks, but there is the obvious 8th item that is not marked complete. I will say that it's possible to have a parent task complete with subtasks not complete and that's handled fine. The main distinction here is the count displayed.

I'll play around with the API directly to see if can reproduce the task that way.

[kenliu]

FYI the way the sync process works like this:

1. First calls the Todoist API and fetches all of your project/task data at once
2. Parses the JSON response from the API server and populates a Store struct (defined in sync.go).
3. This struct is then marshalled back to JSON and stored on your local filesystem in the .cache directory.

The crash is occurring in step 2.

[kenliu]

I will say that it's possible to have a parent task complete with subtasks not complete and that's handled fine.

To clarify you mean that it's handled fine by the official web clients and not by todoist cli, right :)

[HacDan]

I do not actually. I was able to process a sync without issue with the CLI. I didn't test on 22.04, but I'm sitting on a 20.04 machine currently and sync occurred without issue. I believe the issue here to be a technically impossible state from official clients.

[kenliu]

It's possible that this wasn't handled properly in the past by the official clients and changed more recently. Another possible cause is using the todoist CLI to change the state of subtasks -- I haven't checked to see how the CLI handles closing tasks with open subtasks, that could be a bug too.

[jeroenmimpen]

Some extra information, in case this might be useful: I have only used the API/CLI for adding tasks, never for completing. So this completion was done via either the official web-client or the (iphone) app. Also: this list is shared with several other todoist accounts. This particular item was added on '1 Dec 2021 · 10:09 AM' and probably completed a few days later (most likely by some other todoist-account with whom this project was shared, I can check, but the 'load more history' button goes back 1 week at a time, so I would rather not since that is going to take some time😁).

[HacDan]

Even when using Curl directly with the API I was not able to reproduce this exact scenario. I'd say your best bet for resolving this is to check off that task or delete it from the grocery list. This is hopefully a one-off situation.

[jeroenmimpen]

I'd say your best bet for resolving this is to check off that task or delete it from the grocery list. This is hopefully a one-off situation.

I did, and I encountered 3 more cases, the last 'completed' list even had 2 uncompleted items (20 completed, 2 uncompleted). I completed all the tasks. And now things are working again for me.

Thanks for the help!

[kenliu]

My guess is that this is due to some behavior that was there in the past (back in 2021) that was since fixed by Todoist. It probably wouldn't hurt for us to add some assertions in the code to check for this so that if it does happen to other users they know what to do -- there are a couple of other bug reports here in GitHub where people reported sync issues, and I wonder if they have the same cause.