Library scan fails on SFTP mounted storage

[lyn1337]

Behaviour:

• Music app offers to scan new entries
• Scan is very slow, processes 10 at a time
• Stops after a short while with progress unchanged until page refresh

Debug logs:

  Error    PHP                Undefined array key "mpeg" at           2024-07-20T19:27:27+00:00 
                              /srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731                           

  Debug    music              Scanning 10 files of user               2024-07-20T19:27:30+00:00 
                              me                                                  

  Debug    music              Scanning 10 files of user               2024-07-20T19:27:34+00:00 
                              me                                                  

  Debug    music              Scanning 10 files of user               2024-07-20T19:27:38+00:00 
                             me                                                  

  Debug    music              Scanning 10 files of user               2024-07-20T19:27:42+00:00 
                              me                                                  

  Debug    music              Scanning 10 files of user               2024-07-20T19:27:46+00:00 
                              me                                                 

  Debug    no app in context  Logging out                             2024-07-20T19:27:48+00:00 

  Debug    no app in context  Session token invalidated before logout 2024-07-20T19:27:48+00:00 

  Error    PHP                Trying to access array offset on value  2024-07-20T19:27:48+00:00 
                              of type null at                                                   
                              /srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731                           

  Error    PHP                Trying to access array offset on value  2024-07-20T19:27:48+00:00 
                              of type null at                                                   
                              /srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731                           

  Error    PHP                Undefined array key "mpeg" at           2024-07-20T19:27:48+00:00 
                              /srv/nextcloud/apps/music/3rdparty/getID3/getid3/module.audio.mp3.php#1731                           

  Error    index              OCP\Files\NotFoundException:            2024-07-20T19:27:48+00:00 
                              /me/files/n at                             
                              lib/private/Files/Node/Root.php line                              
                              207                                                               
                                                                                                
                               0. .../LazyUserFolder.php line 72                                
                                  OC\Files\Node\Root->get(                                      
                                                                                                
                                  )                                                             
                               1. .../Util.php line 332                                         
                                  OC\Files\Node\LazyUserFolder->get(                            
                                                                                                
                                  )                                                             
                               2. .../LibrarySettings.php line 107                              
                                  OCA\Music\Utility\Util::getFolderFromRelativePath(                           
                                                                                                
                                  )                                                             
                               3. .../Scanner.php line 514                                      
                                  OCA\Music\Utility\LibrarySettings->getFolder(                           
                                                                                                
                                  )                                                             
                               4. .../ApiController.php line 213                                
                                  OCA\Music\Utility\Scanner->scanFiles(                           
                                                                                                
                                  )                                                             
                               5. .../Dispatcher.php line 230                                   
                                  OCA\Music\Controller\ApiController->scan(                           
                                                                                                
                                  )                                                             
                               6. .../Dispatcher.php line 137                                   
                                  OC\AppFramework\Http\Dispatcher->executeController(                           
                                                                                                
                                  )                                                             
                               7. .../App.php line 184                                          
                                  OC\AppFramework\Http\Dispatcher->dispatch(                           
                                                                                                
                                  )                                                             
                               8. .../Router.php line 315                                       
                                  OC\AppFramework\App::main(                                    
                                                                                                
                                  )                                                             
                               9. .../base.php line 1069                                        
                                  OC\Route\Router->match(                                       
                                                                                                
                                  )                                                             
                              10. index.php line 39                                             
                                  OC::handleRequest(                                            
                                                                                                
                                  )                                                             

  Debug    music              Scanning 10 files of user               2024-07-20T19:27:48+00:00 
                              me                                                  

[paulijar]

Thanks for the report.

According the timestamps in your log, scanning each 10 files patch has taken about 4 seconds, i.e. 0.4 seconds per file. I don't think this is unreasonable, considering that each file has to be opened and parsed individually and they reside on another server. I'm not familiar with the SFTP external storages, but depending on its implementation, parsing the contents of the files on the storage may require downloading the entire file from the remote server to your OC/NC server. Or if the integration is smart enough, then partial download may be enough. Using internal storage, you should see 5-10 times faster scan speed.

However, the scanning stopping entirely after a while is unexpected. When the scanning has stopped and you reload the page, does the scanning then again work for a short while before stopping again, or do you need to erase the previously scanned files before you can scan again?

Curiously, the log looks like the user had been logged out in the middle of the scanning process and then reading the files started failing (probably because the access to the storage was lost due to the log-out).

One thing to try would be to do the scanning from the command line, and see, if it makes any difference. The command to give in the main folder of owncloud or nextcloud is something like sudo -u www-data php occ music:scan your_user_name.

If everything else fails, then you may also disable the metadata scanning from the Settings view of the Music app. You will miss out on many of the features of the Music app but at least the scanning should be much faster as the contents of the files are not read.

[lyn1337]

Well alright, I'd call it very slow.

After the refresh, it would scan a few more files and then make no more progress. I've tried the same on the command line and got no further.

[paulijar]

Did you test without the metadata scanning? I assume that it would be a lot faster but maybe still wouldn't fix the problem of the scan stopping after a while.

BTW, what version of Nextcloud or ownCloud are you using?

[lyn1337]

BTW, what version of Nextcloud or ownCloud are you using?

Last version of 27.X and then NC 28.0.8.1

Haven't tested without metadata scanning yet; suspect there's something foul with the SFTP storage option on NC.
I've just rsync'd my collection and the scan went over it quick enough.