Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Some Watchlist "Problems" #21

Open
dominikheiss opened this issue Aug 23, 2023 · 63 comments
Open

Some Watchlist "Problems" #21

dominikheiss opened this issue Aug 23, 2023 · 63 comments

Comments

@dominikheiss
Copy link

dominikheiss commented Aug 23, 2023

Hey guys,

I managed to get everything working in my UNRAID environment, despite the fact, that it's throwing errors from the Watchlists others users using my Plex server (does this only work for "local users"?
Also my watchlist is being created, but not copied to cache :(
Don't know where to start sadly...I hope someone can support me :)

image

image
image

image

I also couldn't use my public dns name for my plex server (even valid certificates and stuff, but it told me "token is malformed"...)

@bexem
Copy link
Owner

bexem commented Aug 23, 2023

Hello! I'm sorry you're experiencing some issues...

it's throwing errors from the Watchlists others users using my Plex server

I've recently noticed an access error with the watchlist feature. Upon checking, I found that the updated plexapi now requires the api call to specify when it's using a token, instead of merely passing the token as before.

(does this only work for "local users"?)

No, it fetches all users.

I also couldn't use my public dns name for my plex server (even valid certificates and stuff, but it told me "token is malformed"...)

I only utilize the external domain for Plex. Due to my intentional firewall settings, I can't access Plex using the internal IP from the server running the script. Therefore, any issues related to domain access shouldn't originate from the code but I can't rule it out completely, I base myself on the fact no other user is experiencing it and during my quick review of the code it still uses up-to-date plexapi calls.

Also my watchlist is being created, but not copied to cache

This is odd, I'm guessing from the size of the file you actually have a list of media inside?

Try with the updated code, let me know, if it still doesn't work, run it with debug on and send me the log, I'll try and have a look when possible.

@dominikheiss
Copy link
Author

Thanks for answering that fast ;)

I updated your script, deleted the watchlist_cache and rerun the script with debugging enabled:
plexcache_log_20230823_1941.log: https://pastebin.com/b51U0pcb

plexcache_watchlist_cache.json is filled with my stuff (no other users media) with a correctly edited path. The media can also be found on my server.

The DNS topic does not matter... I was only thinking if that could be related.
I'm using Chronos Docker with a host network adapter and when I add:
"PLEX_URL": "https://plex.mydomain.level/",

I get this:

image

But like I said, I'd be more than happy if the watch list and other users would work ;)

@bexem
Copy link
Owner

bexem commented Aug 23, 2023

With chronos I had this kind of issues, literally randomly, sometimes I just had to reboot the container, sometimes change the network of the container. That's why I stopped using it and advising it as the best way to run the script.

Regarding the error, I've updated the code once again, I'm not sure why it gives you that error.
The script practically fetches all the users in your server, and then iterate each one of them to fetch the watchlist.
The fact the fetches "random_username1" and then says it's not found, it doesn't make sense...
What I thought is the script might have issues with complex usernames and titles, so now it fetches all the users and if they have a username it uses that rather than the title...according to the documentation, it shouldn't matter, but worth a shot.
I've also added a delay, so that it doesn't spam the plexapi when requesting the watchlist for many users.

@dominikheiss
Copy link
Author

Okay, understood.... I think I will change from Chronos to User-Scripts then.

But sadly it still does not catch the other users watchlist....only my data gets populated in the file, BUT also not copied, even though available.

https://pastebin.com/wW4c1tsQ

@bexem
Copy link
Owner

bexem commented Aug 23, 2023 via email

@dominikheiss
Copy link
Author

dominikheiss commented Aug 23, 2023

I just did and sadly it's exactly the same :(
I'm trying some more things but as everything besides watchlists is working I don't think it's a configuration error....but who knows ;)

Thanks for your effort though!

@dominikheiss
Copy link
Author

dominikheiss commented Aug 24, 2023

//UPDATE:
I got my setup working with using the public DNS name. Involved some fiddling with cloudflare tunnel.

Despite that I still get the errors from the watchlists and no media from the watchlist file gets pulled to the cache.
Just tell me if I can support you with troubleshooting ;)

Also one more question: how does the "watched_move" process act, if movieX is on two users decks and one of them watched it. Will it be moved back to the array or only if it is watched from all user's decks?

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

//UPDATE: I got my setup working with using the public DNS name. Involved some fiddling with cloudflare tunnel.

That's great!

Also one more question: how does the "watched_move" process act, if movieX is on two users decks and one of them watched it. Will it be moved back to the array or only if it is watched from all user's decks?

The common media will be moved only when nobody has it in the watchlist or ondeck, when iterating all the media, the script lists all the files that needs to go the array and to the cache, and before actually moving them, it "compares notes" and skips any files destined to the array if the same file is meant to stay or be moved to the array. (Otherwise you will have a ping pong effect, wasting time and energy).

I still get the errors from the watchlists and no media from the watchlist file gets pulled to the cache.

Can you try this?
https://pastebin.com/pctPrGFi

It's the standalone version of the watchlist fetching, it will print the whole fetched media for each user.
It will likely give the same error as it's the same code, but at least debugging should be easier.

@dominikheiss
Copy link
Author

Here is the output of the file:

https://pastebin.com/gDGYCDN9

I tried to troubleshoot with ChatGPT like 2 hours today, but he is no magician it seems ;)

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

Here is the output of the file:

https://pastebin.com/gDGYCDN9

I tried to troubleshoot with ChatGPT like 2 hours today, but he is no magician it seems ;)

Yeah I wish it could fix the (whole) code but unfortunately it isn't as capable yet...

I've implemented an error catching, so for the users it cannot "find" it will still continue but in a clean way.

https://pastebin.com/pctPrGFi

I'm having trouble understanding why this issue is occurring.
My server is accessed by both local users and external users and both the script and the snippet are retrieving all the watchlist media for all my users.
I'm unsure if this is due to a account permission problem or an issue with the APIs themselves.
The script/snippet literally fetched the user list, and uses that information to request the watchlist, but when it does it, plex says that the user doesn't exist? I just can't understand why right now.

@dominikheiss
Copy link
Author

Here is the output: https://pastebin.com/MaTkswQt

Yes I agree, the error is super weird.

But even if we ignore that it runs into the supposed "API errors", it should read the content of the "plexcache_watchlist_cache.json" (which contains my content - with correct path) and move it to the SSD.

But man, if I am the only one with this problem, don't spend too much time with it.
It's more then okay, that the script fetches all the on-deck stuff from every user.

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

It's because it ends with an error and it doesn't proceed, if I manage a decent error catching when is having this issues, it should then continue and copy whatever it managed to fetch from the watchlist(s).

https://pastebin.com/pctPrGFi
I've updated it again, let me know if it still gives error.

@dominikheiss
Copy link
Author

Ah okay...understood.

image

@dominikheiss
Copy link
Author

It can't be an error in my settings.json right?
I mean with all the path variables and stuff..... if these were the problem, it wouldn't move on-deck stuff to the cache, right?

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

Well no, you are having the same user error not found even on my test script which doesn't touch the settings files at all. And I have set a fairly decent settings loader to avoid settings error (at least obvious ones).

The last error you had it's entirely on me, I forgot to paste
from plexapi.exceptions import NotFound
at the beginning of the file on pastebin🤦🏻‍♂️

It should hopefully work now?

@dominikheiss
Copy link
Author

@dominikheiss
Copy link
Author

Look, that's what I meant:

image

plexcache_log_20230824_2025.log

---> without touching other users it shows no errors, but does not proceed processing the watchlist, or am I mistaken?

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

What is the content of the watchlist file?
The settings file seems correct and the logs doesn't show any errors.

I've re-updated the code on pastebin, it includes another error catching for BadRequests (too many requests).

I really gotta make myself work on the rewrite of the script so that debugging and improving the script will be way easier.

@dominikheiss
Copy link
Author

dominikheiss commented Aug 24, 2023

The watchlist file content seems valid...I checked the paths.
https://pastebin.com/DX0L4b9S

The new script is running for 2 minutes, no output yet :)

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

I've checked the files you have on the watchlist_cache file and they are present in the plexcache log file, it means they are in the list of files that needs to be moved. Debug was on, so obviously it wasn't moving anything.

About the other script running for that long...I guess I went too far with the delay time, just stop it.

@dominikheiss
Copy link
Author

dominikheiss commented Aug 24, 2023

Debug was on, so obviously it wasn't moving anything.

Yeah sure, I also did that, but with debug it should calculate the GB of the files it would move. At least that's what I saw in other Debug runs. And even without debug it does not move them :(

Here is the output: https://pastebin.com/GLn5ZkYy

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

So this file
"/mnt/user/data/media/tv/permanent/HD/Carnival Row (2019) [imdb-tt0489974]/Season 1/Carnival Row (2019) - S01E01 - Some Dark God Wakes [WEBRip-1080p][AC3 5.1][DE+EN][x265]-FuN.mkv"

I suppose is not on the cache drive?
If not it should definitely move that file.

I've updated the main script including what I've learned from the pastebin, I've also added a listing of all the cache files at the end of the script, right before moving files to the cache, just to have some extra info.

@dominikheiss
Copy link
Author

dominikheiss commented Aug 24, 2023

Sadly it isn't:

image

It sits here:

image

Log from the updated version:
plexcache_log_20230824_2139.log

Here with Debug enabled:
plexcache_log_20230824_2141.log

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

Okay try now, it should work.....................................

@dominikheiss
Copy link
Author

It really works - at least for my user ;)
The others are still API errored, 401, unauthorized whatever :)

plexcache_log_20230824_2231.log

@bexem
Copy link
Owner

bexem commented Aug 24, 2023

Yeah unfortunately I haven't came up with a solution for that yet...at least the main user works for you...
I'm curious if any other user is experiencing your same issue, either way I will carry on and working on it as well as the zfs issue I've left for a while.

Thank you for your patience :)

@dominikheiss
Copy link
Author

dominikheiss commented Aug 24, 2023 via email

@dominikheiss
Copy link
Author

What is the ZFS issue, if I may ask? (as my cache disk is a raid1 ZFS array ;) )

@dominikheiss
Copy link
Author

dominikheiss commented Aug 25, 2023

I have one more question regarding the watchlist of other users:
Does it really work for non-home users? Or did I misinterpret that?

Because the script uses "switchHomeUser" and my users are "remote".

Other user's on-deck stuff works, because it's stored on my actual server, but their watchlist data is like in their own account, hence I get the weird authentication errors and such.
--> Just an assumption - I have no clue what I'm talking ;)

@bexem
Copy link
Owner

bexem commented Aug 25, 2023

What is the ZFS issue, if I may ask? (as my cache disk is a raid1 ZFS array ;) )

There is another open issue, where another user was having trouble with the script moving files from the array to the cache, cannot remember now they have either the array or the cache in ZFS.
But if the script is working for you, it means it works as it should always have (python can be very platform agnostic)...that's good I guess 😅

I have one more question regarding the watchlist of other users: Does it really work for non-home users? Or did I misinterpret that?

Because the script uses "switchHomeUser" and my users are "remote".

Other user's on-deck stuff works, because it's stored on my actual server, but their watchlist data is like in their own account, hence I get the weird authentication errors and such. --> Just an assumption - I have no clue what I'm talking ;)

When using the Plex API, the "PlexServer" endpoint allows access to the watchlist of the authenticated (main) user. However, it doesn't provide a direct way to access watchlists of other users.
MyPlexAccount allows it instead, but I have to use plex servers and that's why I have to use internet, and the only way to switch users is "switchHomeUser" at least according to the api documentation.
The "on-deck" information is stored on the main server, which is why we can access it without issues. But for watchlists, since the data is account-specific, we're limited to accessing only the main user's watchlist via the "MyPlexAccount" connection.

Either way I have home users and remote users, and the script manages to gather all the watchlist:

Fetching Emanuele Bergamini's watchlist media...
Emanuele Bergamini: /media/tvseries/Strange Planet/Season 1/Strange Planet - S01E01 - The Flying Machine WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Strange Planet/Season 1/Strange Planet - S01E02 - Greyscale Finger Bandit WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Strange Planet/Season 1/Strange Planet - S01E03 - Careful Now WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Strange Planet/Season 1/Strange Planet - S01E04 - The Big Wet WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Strange Planet/Season 1/Strange Planet - S01E05 - Family, Fandom, Footorb WEBDL-1080p.mkv
Emanuele Bergamini: /media/movies/The Orphanage (2007)/The Orphanage (2007) Bluray-1080p.mkv
Emanuele Bergamini: /media/tvseries/Future Man/Season 1/Future Man - S01E01 - Pilot WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Future Man/Season 1/Future Man - S01E02 - Herpe - Fully Loaded WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Future Man/Season 1/Future Man - S01E03 - A Riphole in Time WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Future Man/Season 1/Future Man - S01E04 - A Fuel's Errand WEBDL-2160p.mkv
Emanuele Bergamini: /media/tvseries/Future Man/Season 1/Future Man - S01E05 - Justice Desserts WEBDL-2160p.mkv
Fetching Guest's (LOCAL/HOME USER) watchlist media...
Guest: /media/movies/Insidious Chapter 5 (2023)/Insidious The Red Door (2023) WEBDL-2160p.mkv
Fetching REDACTED_REMOTEUSER's watchlist media...
Fetching REDACTED_REMOTEUSER's watchlist media...
Fetching REDACTED_REMOTEUSER's watchlist media...
REDACTED_REMOTEUSER: /media/movies/Elemental (2023)/Elemental (2023) WEBDL-2160p.mkv
REDACTED_REMOTEUSER: /media/movies/Life Is Beautiful (1997)/Life Is Beautiful (1997) Bluray-1080p.mkv
REDACTED_REMOTEUSER: /media/movies/The Super Mario Bros. Movie (2023)/The Super Mario Bros. Movie (2023) WEBDL-2160p.mkv
REDACTED_REMOTEUSER: /media/movies/Guardians of the Galaxy Vol. 3 (2023)/Guardians of the Galaxy Vol. 3 (2023) WEBDL-2160p.mkv
REDACTED_REMOTEUSER: /media/movies/Transformers Rise of the Beasts (2023)/Transformers Rise of the Beasts (2023) WEBDL-2160p Proper.mkv
REDACTED_REMOTEUSER: /media/movies/Terminator 2 Judgment Day (1991)/Terminator 2 Judgment Day (1991) Bluray-2160p.mkv
REDACTED_REMOTEUSER: /media/movies/Ghostbusters Afterlife (2021)/Ghostbusters Afterlife (2021) WEBDL-2160p.mkv
SCR-20230825-oyqx-2 SCR-20230825-oxyu-2

At least those are what I understood to be remote users...

@bexem
Copy link
Owner

bexem commented Aug 25, 2023

Apologies, I should have tested it...I've changed how I passed the variable and it seems to be working fine on my end!
Check the last commit!

@dominikheiss
Copy link
Author

dominikheiss commented Aug 25, 2023

No problem ;)

The variable error is gone, but now we have the same situation as before:

plexcache_log_20230825_2005.log

image

@bexem
Copy link
Owner

bexem commented Aug 25, 2023

All the errors you mean?
https://pastebin.com/0DBjrFUp

This an extracted and improved version of the watched fetching...It will fetch all users and should handle unauthorised users like the watchlist one.

@dominikheiss
Copy link
Author

Naaah stop. It is soooooo weird ;)

I don't know why, but I just did another Debug-Run and now it has populated the watched file (which it did not the last time) and told me that it would move 100GB to the array.
Then I started a non debug-run and now it moves 71GB to the array ;)

But doesn't matter, it moves something back^^

@bexem
Copy link
Owner

bexem commented Aug 25, 2023

So it works...odd about the two different numbers but I guess is what plex told to the script...Anyways the improved version should not affect much, if you don't mind testing it anyway before I commit it?

@dominikheiss
Copy link
Author

Yeah, works!! Thank you very much!

I will test it asap... just need to wait for the 71GB transfer.

@bexem
Copy link
Owner

bexem commented Aug 25, 2023

So I've tested it quite few times now and improved it a little more, I've also submitted it in the last commit, so you can either try the script directly or the pastebin one, they are virtually the same.
By the time you run them, you probably don't have any more watched media to move 😅 (I am also copying 98GB, I do have a longer cache expiry time than most)

@dominikheiss
Copy link
Author

haha....yeah took quite a while.

but I also tested the new version and it worked. Ty ;)

@bexem
Copy link
Owner

bexem commented Aug 25, 2023

Perfect, I'm glad it works.
Still baffled about your users situation I will keep investigating!

@dominikheiss
Copy link
Author

dominikheiss commented Aug 25, 2023 via email

@dominikheiss
Copy link
Author

You are also using Unraid, right?

I have the issue, that ur script is running with root privileges and messing with my permissions in the copy process ;)
Do you know how I can keep the "nobody" owner of files/folders with your script?

@bexem
Copy link
Owner

bexem commented Aug 26, 2023

I am using unraid as well, and I did encounter some issues with the permissions but the script is not the only one messing around with it and I have a secondary script to make sure the permissions are always correct in the media folders...

But anyway you are right and the script should not mess with the permissions and rather keep them as they are.

I've updated the code, I've quickly run it but because there is nothing to move I can't tell you for sure how well it works.
Might not be relevant for you, but for anyone else reading, it should handle windows as well.

@dominikheiss
Copy link
Author

Thanks for the fast response and help - seems to work as it should ;)
Have a nice weekend!!

@bexem
Copy link
Owner

bexem commented Aug 30, 2023

Hey there, I hope everything is still working fine! Unfortunately I've not managed to fix your users error, but another user on reddit was having some issues with the folder creation in the version of the script we have been editing (and you, probably still using).
Just wanted to let you know I've updated it to fix it.

I have also implemented a duplicate check, so it will remove the media file from the source if already present in the destination, avoiding back and forth file moves. I will implement a setting for it eventually!

@dominikheiss
Copy link
Author

Hey, yeah everything's cool, thanks for asking ;)
Thanks for the update, will test it asap!

@dominikheiss
Copy link
Author

dominikheiss commented Aug 30, 2023

Hey, I don't know if it's the new version or some randomness, but I got an error, which prevents the function itsself.
plexcache_log_20230830_1827.log

image

I disabled watchlist for now, but does not change anything.

@bexem
Copy link
Owner

bexem commented Aug 30, 2023

Uhm that's odd, just to be sure, the script you are using, when did you get it? Because I did commit 5h ago, just want to be sure it is the latest one you are using

@dominikheiss
Copy link
Author

dominikheiss commented Aug 30, 2023 via email

@bexem
Copy link
Owner

bexem commented Aug 30, 2023

If you don't mind testing the latest one as I did change some stuff and fixed others, hopefully it fixes your new issues as well...

@bexem
Copy link
Owner

bexem commented Aug 30, 2023

If you don't mind testing the latest one as I did change some stuff and fixed others, hopefully it fixes your new issues as well...

I've reversed some changes, just updated the repository, it did start giving the same issues on my end too (even tough I run it like 5 times with me playing around with files and permissions earlier and it gave me no issues)...

@dominikheiss
Copy link
Author

Hey, I just ran the updated version and got the same error with the same file :/
I try to delete watched-file and run again

@dominikheiss
Copy link
Author

Now it moves stuff to the array, after deleting watched-cache.
I don't know what's up :)

@bexem
Copy link
Owner

bexem commented Aug 30, 2023

might have put the wrong paths in the cache file. Let me know if it continue to work. Otherwise this the same version you were using before updating at all: https://pastebin.com/w97WAjk4
Sorry about this, it was literally working fine for me and also another user, then it started giving issues, and because my server is just crashing I can't even test properly anymore!
Sorry again!

UPDATE: I've, once again, committed another update, just because the permissions were not set consistently all the time. Now it should and appears so judging by my testing and another user! 🎉

@dominikheiss
Copy link
Author

I can also confirm the new version is working!

@dominikheiss
Copy link
Author

Mhhh... sadly I got the same error again.
Had to remove the watched-cache again and then it worked again ;)
Is it only me?

@bexem
Copy link
Owner

bexem commented Sep 1, 2023

Mhhh... sadly I got the same error again.
Had to remove the watched-cache again and then it worked again ;)
Is it only me?

Is it exactly the same error as your last log?
Error checking free space and moving media files to the cache: [Errno 2] No such file or directory:
(I am aware "to the cache" in this case is actually meant to say "to the array", I alway forget to fix that text)

What permissions does the Show and Season folder have? And, when removing the watched cache file it would actually move that same file with no issues?

Also (I need to start using version number inside the script), considering your previous comment, you are using the latest commit, right?

@dominikheiss
Copy link
Author

dominikheiss commented Sep 2, 2023

Yeah, was the same error as you mentioned:
2023-09-01 17:45:16,278 - CRITICAL - Error checking free space and moving media files to the cache: [Errno 2] No such file or directory: '/mnt/user/data/media/movies/temporary/HD/The Machine (2023)/The Machine (2023)-Bluray-1080p-x264 AC3[DE+EN] [EN+DE].mkv'

The permissions are "nobody" as owner and rwx as actual permission.
Yeah right, after I deleted the watched-cache it works.
-> the empty folder stayed on the cache though....which is also weird.

I had the latest revision of yours.

@bexem
Copy link
Owner

bexem commented Sep 2, 2023

The permissions appear correct then.
About the folder, to be fair I never thought of that, I've always focused on moving the file rather, leaving the folders behind, should probably sort that out.

I did a commit last night (~10h ago), did not change much, just the order on when the script resets its umask to then give the right permissions to files and folders and, finally, the "to the array" incorrect text. Unfortunately it is not tested as my cache drive is sort of dead, so I can't run my own script currently ☹️

@dominikheiss
Copy link
Author

Oh okay, dead cache drive sucks.
I really don't know where to search for a solution.
I think I have to disabled the "wachted-cache" feature, as it only works when it recreates it. When it is present I get the "CRITICAL - Error checking free space and moving media files to the array" message.

@bexem
Copy link
Owner

bexem commented Sep 2, 2023

I will definitely take a look at it. It's odd that I didn't make any changes to the watched media or the caching system. The last time I used the script, it was working okay. Additionally, the other active user (there might be more, I hope) of the script hasn't reported any issues either.

Nevertheless, when I come around to rewriting the script, I'll ensure that it's handled better, I suppose. At least it functions without the cache. To be fair, since it's fetched locally, it shouldn't significantly impact the run time of the script.

The watchlist, on the other hand, could have been more troublesome without cache. The funny part is that they use the same functions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants