Playback of recordings is often very slow or results in an error

[shred86]

When I try to play a video either from the events page or camera recordings, I often get the spinning loading wheel for a period of time followed by the error: "The media could not be loaded, either because the server or network failed or because the format is not supported." However, sometimes it seems to work fine where I'll see the spinning loading wheel for maybe 3-5 seconds and the recordings play just fine.

I'm on the latest version of Frigate (0.11.1) running on an Odyssey X86 Blue J4125 with an external Western Digital USB 3.0 hard drive (2TB 5400 RPM) mounted to Debian for the Frigate docker container to use. Frigate is the only application that uses this external hard drive and I've confirmed it's plugged into a USB 3.0 port on the Odyssey.

Using hdparm, the read times are showing around 37 MB/sec, which seems slow. I have another 2.5" HDD mounted for backups and that one reads around 129 MB/sec and the M.2 SSD reads around 350 MB/sec. The Odyssey is also hard wired (1GB NIC), so I don't think network speeds is the issue. As mentioned above, it sometimes will work fine but other times results in a spinning wheel for an extended period of time followed by the error message.

Nothing showing up in my Frigate logs, but I'm just running the default logging settings.

Any tips/recommendations on how to resolve this issue?

My first thought is trying to figure out why the external HDD read speeds are so slow. Perhaps that's normal for USB 3.0 and a spinning 5400 RPM drive? I have four cameras setup with Frigate, four of which are recording 24/7 at 4MP. This is on the latest version of Debian so it's running the latest USB drivers.

Second thought is if the slow read speeds are about what I can expect for this setup, then maybe switching Frigate over to the internal 2.5" HDD with better read performance.

Edit:

Here's the results of sudo hdparm -I /dev/sdc:

ATA device, with non-removable media
	Model Number:       WDC WD20SDRW-11VUUS1                    
	Serial Number:      redacted
	Firmware Revision:  01.01A01
	Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
	Used: unknown (minor revision code 0x006d) 
	Supported: 10 9 8 7 6 5 
	Likely used: 10
Configuration:
	Logical		max	current
	cylinders	16383	0
	heads		16	0
	sectors/track	63	0
	--
	LBA    user addressable sectors:   268435455
	LBA48  user addressable sectors:  3906963632
	Logical  Sector size:                   512 bytes
	Physical Sector size:                  4096 bytes
	Logical Sector-0 offset:                  0 bytes
	device size with M = 1024*1024:     1907697 MBytes
	device size with M = 1000*1000:     2000365 MBytes (2000 GB)
	cache/buffer size  = unknown
	Form Factor: 2.5 inch
	Nominal Media Rotation Rate: 5400
Capabilities:
	LBA, IORDY(can be disabled)
	Queue depth: 32
	Standby timer values: spec'd by Standard, with device specific minimum
	R/W multiple sector transfer: Max = 16	Current = 16
	Advanced power management level: 128
	DMA: *mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 udma6 
	     Cycle time: min=120ns recommended=120ns
	PIO: pio0 pio1 pio2 pio3 pio4 
	     Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
	Enabled	Supported:
	   *	SMART feature set
	    	Security Mode feature set
	   *	Power Management feature set
	   *	Write cache
	   *	Look-ahead
	   *	WRITE_BUFFER command
	   *	READ_BUFFER command
	   *	NOP cmd
	   *	DOWNLOAD_MICROCODE
	   *	Advanced Power Management feature set
	    	Power-Up In Standby feature set
	   *	SET_FEATURES required to spinup after power up
	   *	48-bit Address feature set
	   *	Mandatory FLUSH_CACHE
	   *	FLUSH_CACHE_EXT
	   *	SMART error logging
	   *	SMART self-test
	   *	General Purpose Logging feature set
	   *	64-bit World wide name
	   *	IDLE_IMMEDIATE with UNLOAD
	   *	{READ,WRITE}_DMA_EXT_GPL commands
	   *	Segmented DOWNLOAD_MICROCODE
	   *	Gen1 signaling speed (1.5Gb/s)
	   *	Gen2 signaling speed (3.0Gb/s)
	   *	Gen3 signaling speed (6.0Gb/s)
	   *	Native Command Queueing (NCQ)
	   *	Host-initiated interface power management
	   *	Phy event counters
	   *	Idle-Unload when NCQ is active
	   *	NCQ priority information
	   *	READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
	    	DMA Setup Auto-Activate optimization
	    	Device-initiated interface power management
	    	Software settings preservation
	   *	SMART Command Transport (SCT) feature set
	   *	SCT Write Same (AC2)
	   *	SCT Features Control (AC4)
	   *	SCT Data Tables (AC5)
	    	unknown 206[12] (vendor specific)
	    	unknown 206[13] (vendor specific)
	    	unknown 206[14] (vendor specific)
	   *	Extended number of user addressable sectors 
	   *	Device encrypts all user data
	   *	DOWNLOAD MICROCODE DMA command
	   *	WRITE BUFFER DMA command
	   *	READ BUFFER DMA command
	   *	Data Set Management TRIM supported (limit 10 blocks)
	   *	Deterministic read data after TRIM
Security: 
	Master password revision code = redacted
		supported
	not	enabled
	not	locked
	not	frozen
	not	expired: security count
		supported: enhanced erase
	382min for SECURITY ERASE UNIT. 2min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: redacted
	NAA		: 5
	IEEE OUI	: 0014ee
	Unique ID	: redacted
Checksum: correct

Results from using iperf3 from my iMac (5GHz Wifi, 80 MHz spacing) to my Odyssey (hardwired 1GB NIC):

[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  47.6 MBytes   398 Mbits/sec                  
[  5]   1.00-2.00   sec  49.0 MBytes   412 Mbits/sec                  
[  5]   2.00-3.00   sec  47.4 MBytes   398 Mbits/sec                  
[  5]   3.00-4.00   sec  50.3 MBytes   422 Mbits/sec                  
[  5]   4.00-5.00   sec  46.4 MBytes   389 Mbits/sec                  
[  5]   5.00-6.00   sec  48.7 MBytes   408 Mbits/sec                  
[  5]   6.00-7.00   sec  48.0 MBytes   403 Mbits/sec                  
[  5]   7.00-8.00   sec  47.5 MBytes   398 Mbits/sec                  
[  5]   8.00-9.00   sec  46.1 MBytes   387 Mbits/sec                  
[  5]   9.00-10.00  sec  48.7 MBytes   409 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   480 MBytes   402 Mbits/sec                  sender
[  5]   0.00-10.04  sec   478 MBytes   400 Mbits/sec                  receiver

[blakeblackshear]

Two things would be helpful to know:

1. Open dev tools and watch the network tab. Which request specifically is taking a long time?
2. If you refresh and try to play the same event or recordings again immediately after the first time, is it still slow?

[NickM-27]

That seems to imply it may be the harddrive causing issues since that's the endpoint that reads the files.

Is there any correlation between which works and which doesn't and the camera?

[shred86]

Unfortunately there isn't. It happens with all of the camera recordings. Sometimes I'll just refresh the browser and try again on the same clip and it works. Other times, I'll try 4-5+ times and end up with the same error, so I usually just try again at a later date/time and it will work.

[blakeblackshear]

If you try the same master.m3u8 url after it was once successful, does it load quickly the second time?

[shred86]

Just to make sure I'm doing what you're asking correctly - if I play a video clip and it's successful, I'll play another video clip (which is also successful) then go back and play the first video clip again. It does load quickly the second time (less than a second).

[blakeblackshear]

That's what I would expect based on what we have seen elsewhere. The nginx VOD module reads the metadata from the segments to make that playlist which requires hitting every file on disk. It caches this metadata for some period of time, which is why the second load is fast. This is why I want to look at adjusting the way we use the VOD module or abandon it.

[flaektrem]

Same problem for me. Even with fresh install (0.11 and 0.11.1) with new DB and clean media folder.
Video never starts, just loading and loading.
If I download clip it works great.
No issues at all with latest 0.10 beta.

@blakeblackshear is there any related changes in 0.11 that can cause this problem?

[blakeblackshear]

If you have a fresh db and this always happens, then you don't have the same issue.

[shred86]

Still having this issue so I just swapped Frigate to use the internal 2.5" HDD instead of the external USB HDD. So far it's running much better but I'll keep an eye on it and post an update as the issue before seemed to vary (sometimes it seemed okay, and other times recordings would just not play all day).

[LaurenceGough]

It seems Frigate is very HDD sensitive, my performance is terrible over an older NAS on a 1Gb short link, granted it is a very old NAS. I usually have to wait a minute or more to load a small 20 second clip, and seeking takes ages too, but I understand others with faster storage are able to load it in under a second.
Although it would be my only complaint about Frigate, but I cannot really complain about it!!

I am a big fan of SSDs but with constant writing from recording it just doesn't make sense to me, nor is it that cost effective currently.

I do need to sort something out though and abandon the NAS I think...

I am too starting to see loading failures occasionally. Hard refreshing or trying another device doesn't fix the issue. I had one earlier, after about 10 minutes it let me load it... weird.

With this one the recording had disappeared... It wasn't in progress either it was a finished one of a person.

[NickM-27]

I mean, I think any software that depends on reading recordings metadata and serving it would struggle on hardware like that. Frigate can absolutely work great on an HDD if it is a decent speed and has cache.