DB Optimizations

[NickM-27]

• Auto vacuum won't defragment the DB but it removes dead pages. For my 1.5 year old DB it reduced the size from 298 MiB to 232MiB
• Optimize queries to remove != which can not use indexes
• Add index for storage calculations
• Increase DB write timeout
• Set synchronous mode to NORMAL
• Increase DB cache

[netlify]

✅ Deploy Preview for frigate-docs canceled.

Name	Link
🔨 Latest commit	ec8f8d5
🔍 Latest deploy log	https://app.netlify.com/sites/frigate-docs/deploys/6481ee7d5829e70008d3908f

[blakeblackshear]

Let's take a look at how homeassistant does this in core.

[NickM-27]

Let's take a look at how homeassistant does this in core.

https://www.home-assistant.io/integrations/recorder/

Automatically purge the database every night at 04:12 local time. Purging keeps the database from growing indefinitely, which takes up disk space and can make Home Assistant slow. If you disable auto_purge it is recommended that you create an automation to call the recorder.purge periodically.

Automatically repack the database every second sunday after the auto purge. Without a repack, the database may not decrease in size even after purging, which takes up disk space and can make Home Assistant slow. If you disable auto_repack it is recommended that you create an automation to call the recorder.purge periodically. This flag has no effect if auto_purge is disabled.

The purge seems to be when they actually remove the data from the DB

https://github.com/home-assistant/core/blob/c67f32b924d7c3b1960149eb5499568aefa14bc7/homeassistant/components/recorder/purge.py

The repack is just a call to VACUUM

https://github.com/home-assistant/core/blob/c67f32b924d7c3b1960149eb5499568aefa14bc7/homeassistant/components/recorder/repack.py#L23-L29

[blakeblackshear]

Should we add a call to VACUUM in this PR or incorporate that later?

[NickM-27]

The changes are causing problems with the tests. I don't think there is an issue doing the indexes manually in SQL especially since this is how all previous indexes have been done as well.

[blakeblackshear]

You have to be cautious about using the model objects for migrations because the model objects change over time. This can cause situations where historical migrations would be modified.

For example:

1. A field is added to a model in migration 002.
2. That field is later removed from the model in migration 007.

Because you removed the field from the model definition, migration 002 is now broken.

[skrashevich]

Because you removed the field from the model definition, migration 002 is now broken.

That's why in the changes I proposed, field names were strings, not objects

[blakeblackshear]

That's why in the changes I proposed, field names were strings, not objects

It should work then. We ran into some issues when we tried to use the model objects to create new tables in the past.

[NickM-27]

The tests were failing on the unique field inside of peewee migrator.py regardless of what I put

[skrashevich]

The tests were failing on the unique field inside of peewee migrator.py regardless of what I put

Can I have more details?

[NickM-27]

The tests were failing on the unique field inside of peewee migrator.py regardless of what I put

Can I have more details?

With what was suggested above this error occurs:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/router.py", line 194, in run_one
    migrate(migrator, self.database, fake=fake)
  File "<string>", line 31, in migrate
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/migrator.py", line 128, in wrapper
    return method(migrator, model, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/migrator.py", line 348, in add_index
    field.unique = unique
AttributeError: 'NoneType' object has no attribute 'unique'

When I follow the peewee docs, this occurs:

[2023-06-08 07:05:37] peewee_migrate                 ERROR   : Migration failed: 017_update_indexes
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/router.py", line 194, in run_one
    migrate(migrator, self.database, fake=fake)
  File "<string>", line 31, in migrate
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/migrator.py", line 127, in wrapper
    return method(migrator, migrator.orm[model], *args, **kwargs)
KeyError: 'recordings'

[skrashevich]

With what was suggested above this error occurs:

Just remove the third option (unique=False) from add_index call.
Strange error, but anyway unique=False is default

[NickM-27]

With what was suggested above this error occurs:

Just remove the third option (unique=False) from add_index call. Strange error, but anyway unique=False is default

I tried that as well, and it also does not work

Traceback (most recent call last):
  File "/workspace/frigate/frigate/test/test_storage.py", line 24, in setUp
    router.run()
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/router.py", line 224, in run
    done.append(self.run_one(mname, migrator, fake=fake, force=fake))
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/router.py", line 194, in run_one
    migrate(migrator, self.database, fake=fake)
  File "<string>", line 31, in migrate
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/migrator.py", line 128, in wrapper
    return method(migrator, model, *args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/peewee_migrate/migrator.py", line 348, in add_index
    field.unique = unique
AttributeError: 'NoneType' object has no attribute 'unique'