utils of history with sqlite

[fj0r]

• backup
• restore
• timing
• top

[fdncred]

it would be good to check for sqlite3 before using it. maybe with the which command.

[fdncred]

There's nothing special about this command. Updates can be done with query db, right?

[fj0r]

It seems that when the database file does not exist, an error will occur when using query db, and using sqlite3 will create a new database. I am not sure. I need to test it.

[fj0r]

 open test.sqlite | query db 'create table test(a text);' 
Error: nu::shell::file_not_found

  × File not found
   ╭─[entry #1:1:6]
 1 │ open test.sqlite | query db 'create table test(a text);' 
   ·      ─────┬─────
   ·           ╰── file not found
   ╰────
  help: test.sqlite does not exist

 'create table test(a text);' | sqlite3 test.sqlite
# After using sqlite3 to create the database, you will find that the table already exists using `query db`
 open test.sqlite | query db 'create table test(a text);'                                                                              
Error:   × Failed to query SQLite database
   ╭─[entry #1:1:29]
 1 │ open test.sqlite | query db 'create table test(a text);' 
   ·                             ──────────────┬─────────────
   ·                                           ╰── table test already exists in create table test(a text); at offset 13
   ╰────

[fj0r]

I don't know how to create an empty database file using query db, ever used this function before

export def empty-sqlite [] {
    # sqlite3 empty.db "VACUUM;"; cat empty.db | gzip | encode base64
    'H4sIAAAAAAAAAwsO9MksSVVIyy/KTSxRMGYQYGBkZHBQUGBgYGCEYhhAZhMLGBn0ihbwglgCZOgeBaNgFIyCUTAKRsEoGAWjYBSMglEwCkYBVQAANHgbMAAQAAA='
    | decode base64 --binary | gzip -d
}

[fj0r]

I'm very sorry, I saw that the reply had a Pending tag, and I found that I need to submit it. I don't know this before.

[fdncred]

I've been wondering if the stor family of commands can be used to replace the sqlite3 stuff here. For backup and restore, it seems like all you'd have to do is stor import --file-name $nu.history-path and then stor export --file-name /some/other/path/history.sqlite3. Have you tried that yet?

[fj0r]

It seems feasible, but the database file is larger than the sql file, it is acceptable.
The main question is, if other applications are also using stor and create tables, will stor import delete the existing tables, and if there are others tables when saving, will they be saved together?

[fj0r]

If it is not limited to text format, can it just copy the database file?
Text formats can be parsed and processed, and there may be some scenarios where this is required.

[fj0r]

There seems to be a small problem with stor

stor create -t foo -c { bar: str }
stor insert -t foo -d {bar: "foobar"}
stor insert -t foo -d {bar: "foo'bar"}
Error:   × Failed to open SQLite connection in memory from insert
   ╭─[source:1:1]
 1 │ nu
   · ▲
   · ╰── near "bar": syntax error in INSERT INTO foo ( bar) VALUES ( 'foo'bar') at offset 37
   ╰────

stor insert -t foo -d {bar: "foo\'bar"}
Error:   × Failed to open SQLite connection in memory from insert
   ╭─[source:1:1]
 1 │ nu
   · ▲
   · ╰── near "bar": syntax error in INSERT INTO foo ( bar) VALUES ( 'foo'bar') at offset 37
   ╰────

stor insert -t foo -d {bar: "foo\\'bar"}
Error:   × Failed to open SQLite connection in memory from insert
   ╭─[source:1:1]
 1 │ nu
   · ▲
   · ╰── near "bar": syntax error in INSERT INTO foo ( bar) VALUES ( 'foo\'bar') at offset 38
   ╰────

[fdncred]

I think this is an interesting set of custom commands. However, it would be nice if they were cross platform. In order to do that, we'd need to use nushell built-in command in place of externals. It would also be good to eliminate the use of the sqlite3 command if possible.

[maxim-uvarov]

I'm sorry for commenting here, but I don't know how to reach @fj0r otherwise.

@fj0r, would you please share why you decided to remove the backup and restore commands from here? Were they erroneous?

[maxim-uvarov]

I'm sorry once again.
I've just found the answer in your comments about using stor.

[fj0r]

@maxim-uvarov https://github.com/fj0r/nushell/blob/main/scripts/history-utils/backup.nu
This part is separated into a file in my personal configuration file repository
Although in general, a simple cp is enough for backup purposes.
By the way: if backup is important, it is best to use a professional tool, such as https://rustic.cli.rs. And:

• Scheduled or automatically triggered
• Incremental backups
• A proper retention strategy
• Back up to object storage

[fj0r]

https://github.com/fj0r/nushell/blob/rustic/scripts/rustic/mod.nu
There's even a completion for rustic, just some basic functionality until I realized that the output of --help should be parsed. But I don't have the time and wanted to see if someone has done it first.

[maxim-uvarov]

@fj0r thank you!

Just as a remark - I once tried updating history with into sqlite and it completely broke my history. I attempted to restore it from the commit with files in $nu.default-config-dir but it didn't work. I don't remember all the details, but I spent like an hour or two restoring history without any changes applied. Though I was much more of a noob back then.

If I understand it correctly, now I should use query db with update statements... Or not?

Still, altering history is on my list, and I will need to learn how to deal with backups.
For now, I still just create commits of the $nu.default-config-dir without ever checking if they are restorable.

[fj0r]

@maxim-uvarov sqlite may damage the database during concurrent writing. Unfortunately, in the nushell environment, the timing of writing to history's sqlite is uncontrollable. For example, if you execute an update statement through sqlite or query db (not sure if query db is locked), if you are operating a history database, when you press Enter, nushell itself will modify the database, and your sql statement will also modify the database, which may be a problem.

UPDATE: after testing, it was found that nushell writes history records first and then executes commands. If there is only one nushell, there is no problem. However, consider this situation. You update all history records in one nushell process. Assuming it takes 10 seconds. Before it is finished, you execute any command in another nushell process. This also requires modifying the database file, which may cause problems.

From the user's perspective, the history of nushell itself is only a conventional history, and some modern environment experiences are not taken into account, such as the search and synchronization of mixed history of multiple machines. If you have higher requirements in this regard, consider https://github.com/atuinsh/atuin

[fdncred]

We run the sqlite history in WAL mode so there shouldn't be any synchronization issues.