Database disk image is malformed #5252

devos50 · 2020-03-27T10:25:37Z

From the error reporter (7.5.0-RC2):

Traceback (most recent call last):
File "src\tribler-gui\tribler_gui\core_manager.py", line 106, in on_received_state
RuntimeError: database disk image is malformed
--LONG TEXT--
Traceback (most recent call last):
File "src\run_tribler.py", line 95, in start_tribler
File "src\tribler-core\tribler_core\session.py", line 440, in start
File "src\tribler-core\tribler_core\session.py", line 170, in load_ipv8_overlays
File "ipv8\attestation\trustchain\community.py", line 64, in __init__
File "ipv8\attestation\trustchain\database.py", line 43, in __init__
File "ipv8\attestation\trustchain\database.py", line 419, in open
File "ipv8\database.py", line 127, in open
File "ipv8\database.py", line 162, in _initial_statements
sqlite3.DatabaseError: database disk image is malformed
--CONTEXT--
{'message': 'Task exception was never retrieved', 'exception': DatabaseError('database disk image is malformed'), 'future': <Task finished coro=<start_tribler_core.<locals>.start_tribler() done, defined at src\run_tribler.py:64> exception=DatabaseError('database disk image is malformed')>}

Related to #1993

qstokkink · 2020-03-27T10:27:23Z

What should we do when the database is corrupt beyond repair? Remove it?

ichorid · 2020-03-27T12:09:45Z

In this case, we should mark the database corrupted and schedule a DB check on the next startup. Ideally, we should employ DB's own mechanisms for that.

devos50 · 2020-04-01T15:27:46Z

@ichorid do we want to fix this issue in 7.5?

ichorid · 2020-04-01T15:38:44Z

There were only 2 bugreports with this kind of bug in the last few months. One should put some real effort to garble their SQLite database to produce this kind of error. It is alson not trivial to recover from it later. I would mark it a low-priority and moved to backlog.

devos50 · 2020-07-20T10:21:23Z

On our (Proxmoxed) Mac, I now have a corrupt metadata.db, which is crashing consistently a few seconds after start-up. It errors with different kind of errors actually, depending on what the interactions with the database are. For example:

Traceback (most recent call last):
  File "ipv8/taskmanager.py", line 131, in done_cb
  File "tribler_core/modules/metadata_store/community/remote_query_community.py", line 149, in on_remote_select_response
  File "tribler_core/modules/metadata_store/store.py", line 287, in process_compressed_mdblob_threaded
  File "concurrent/futures/thread.py", line 57, in run
  File "tribler_core/modules/metadata_store/store.py", line 276, in _process_blob
  File "tribler_core/modules/metadata_store/store.py", line 295, in process_compressed_mdblob
  File "tribler_core/modules/metadata_store/store.py", line 328, in process_squashed_mdblob
  File "<string>", line 2, in process_payload
  File "pony/orm/core.py", line 515, in new_func
  File "tribler_core/modules/metadata_store/store.py", line 409, in process_payload
  File "pony/orm/core.py", line 3974, in get
  File "pony/orm/core.py", line 4077, in _find_one_
  File "pony/orm/core.py", line 4136, in _find_in_db_
  File "pony/orm/core.py", line 938, in _exec_sql
  File "pony/orm/core.py", line 1805, in prepare_connection_for_query_execution
  File "pony/orm/core.py", line 1891, in flush
  File "pony/orm/core.py", line 5393, in _save_
  File "pony/orm/core.py", line 5240, in _save_created_
  File "pony/utils/utils.py", line 106, in throw
pony.orm.core.UnexpectedError: Object ChannelMetadata[new:70] cannot be stored in the database. DatabaseError: database disk image is malformed

I will save this database image.

xoriole · 2020-10-21T16:45:49Z

Adding one more stack trace from the error reporter for this issue

Traceback (most recent call last):
File "src\tribler-gui\tribler_gui\core_manager.py", line 68, in on_core_finished
RuntimeError: The Tribler core has unexpectedly finished with exit code -1073741819 and status: 1!

[PID:7780] 2020-10-22 00:24:14,067 - ERROR <taskmanager:135> RemoteQueryCommunity.done_cb(): Task resulted in error: database disk image is malformed
Traceback (most recent call last):
File "lib\site-packages\pony\orm\dbapiprovider.py", line 55, in wrap_dbapi_exceptions
File "lib\site-packages\pony\orm\dbapiprovider.py", line 279, in execute
sqlite3.DatabaseError: database disk image is malformed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "ipv8\taskmanager.py", line 131, in done_cb
File "src\tribler-core\tribler_core\modules\metadata_store\community\remote_query_community.py", line 149, in on_remote_select_response
File "src\tribler-core\tribler_core\modules\metadata_store\store.py", line 287, in process_compressed_mdblob_threaded
File "lib\concurrent\futures\thread.py", line 57, in run
File "src\tribler-core\tribler_core\modules\metadata_store\store.py", line 278, in _process_blob
File "lib\site-packages\pony\orm\core.py", line 476, in __exit__
File "lib\site-packages\pony\orm\core.py", line 490, in _commit_or_rollback
File "lib\site-packages\pony\orm\core.py", line 381, in commit
File "lib\site-packages\pony\orm\core.py", line 370, in rollback_and_reraise
File "lib\site-packages\pony\utils\utils.py", line 95, in reraise
File "lib\site-packages\pony\orm\core.py", line 379, in commit
File "lib\site-packages\pony\orm\core.py", line 1896, in flush
File "lib\site-packages\pony\orm\core.py", line 5421, in _save_
File "lib\site-packages\pony\orm\core.py", line 5330, in _save_updated_
File "lib\site-packages\pony\orm\core.py", line 949, in _exec_sql
File "lib\site-packages\pony\orm\core.py", line 1784, in reconnect
File "lib\site-packages\pony\utils\utils.py", line 95, in reraise
File "lib\site-packages\pony\orm\core.py", line 947, in _exec_sql
File "<string>", line 2, in execute
File "lib\site-packages\pony\orm\dbapiprovider.py", line 74, in wrap_dbapi_exceptions
pony.orm.dbapiprovider.DatabaseError: database disk image is malformed

(Timestamp: 1603293855257, traceback timestamp: 1603293854068)

7.5.2 7.5.3

ichorid · 2020-10-22T12:56:03Z

Most probably, DB corruption is a result of interrupted/crashed DB upgrade/conversion procedure. Safety measures are disabled doing DB upgrade for speed.

xoriole · 2020-10-22T12:59:52Z

Looking at the uptime and the logs
7.5.2 - 1318.8 seconds
7.5.3 - 179.57 seconds

I'm not sure if it was during upgrade

ghost · 2020-11-03T18:16:45Z

Sentry issue: TRIBLER-3

ichorid · 2020-11-03T20:27:08Z

I'll try fixing it for 7.6. @devos50 , could you please send me the corrupted DB if it is not that big?

devos50 · 2020-11-03T21:03:47Z

@ichorid it seems that I saved that database on our server named "proxmox" and on the VM with ID 105. You can find the corrupt metadata database in the following path: /Users/tribler/tribler-failing-metadata/db/7.5/sqlite/metadata.db.

sentry-for-tribler · 2020-11-05T08:48:58Z

Related error, a user commented: "Even with the tribler update, tribler keeps crashing on my Mac with this error message. It has been this way for the past 4 months and the releases from then. Thanks."
Probably we need to provide an option to re-create the database from scratch for these cases

ichorid · 2020-11-05T09:41:30Z

Probably we need to provide an option to re-create the database from scratch for these cases

thanks for suggestion, @sentry-for-tribler bot! 😉

Jokes aside, I'm going to add a subsystem to schedule a DB check after this kind of crashes.

rgoshko · 2020-11-06T23:28:34Z

Got the same issue today when my Mac mini unexpectedly rebooted:

Traceback (most recent call last):
  File "tribler_gui/core_manager.py", line 123, in on_received_state
RuntimeError: database disk image is malformed
--LONG TEXT--
Traceback (most recent call last):
  File "pony/orm/dbapiprovider.py", line 53, in wrap_dbapi_exceptions
  File "pony/orm/dbapiprovider.py", line 269, in execute
sqlite3.DatabaseError: database disk image is malformed

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run_tribler.py", line 81, in start_tribler
  File "tribler_core/session.py", line 425, in start
  File "tribler_core/modules/metadata_store/store.py", line 135, in __init__
  File "pony/orm/core.py", line 1138, in generate_mapping
  File "<string>", line 2, in check_tables
  File "pony/orm/core.py", line 528, in new_func
  File "pony/orm/core.py", line 1211, in check_tables
  File "pony/orm/dbschema.py", line 78, in check_tables
  File "<string>", line 2, in execute
  File "pony/orm/dbapiprovider.py", line 69, in wrap_dbapi_exceptions
pony.orm.dbapiprovider.DatabaseError: database disk image is malformed

--CONTEXT--
{'message': 'Task exception was never retrieved', 'exception': DatabaseError('database disk image is malformed'), 'future': <Task finished name='Task-1' coro=<start_tribler_core.<locals>.start_tribler() done, defined at run_tribler.py:50> exception=DatabaseError('database disk image is malformed')>}`

I am unsure which db file is "bad", in ~/.Tribler/7.5/sqlite I have the following two db files from today:

32829440 6 Nov 11:28 metadata.db 404717568 6 Nov 11:27 trustchain.db

Thanks.

ichorid · 2020-11-06T23:34:36Z

@rgoshko , thanks for reporting! It is caused by metadata.db. Only that one is handled by Pony ORM in 7.5.x series.

You can try to recover it by creating a dump. Or just delete it (but you will lose your personal channel).

We will add a procedure to handle this kind of failures in 7.6.

rgoshko · 2020-11-09T16:12:09Z

Thanks @ichorid , I removed the metadata.db file and I am back up and running now.

sentry-for-tribler · 2020-11-20T10:38:11Z

Sentry issue: TRIBLER-2F

sentry-for-tribler · 2020-12-30T10:15:32Z

Sentry issue: TRIBLER-55

ichorid · 2021-10-12T10:40:14Z

related to #1993

sentry-for-tribler · 2021-11-22T14:25:33Z

Sentry issue: TRIBLER-9T

sentry-for-tribler · 2021-12-13T10:45:02Z

Sentry issue: TRIBLER-A7

sentry-for-tribler · 2021-12-24T13:43:40Z

Sentry issue: TRIBLER-AE

sentry-for-tribler · 2022-03-07T11:05:17Z

7.11:
Sentry issue: TRIBLER-DJ

sentry-for-tribler · 2022-03-07T11:07:54Z

Sentry issue: TRIBLER-DH

sentry-for-tribler · 2022-12-19T12:17:55Z

Sentry issue: TRIBLER-11R

sentry-for-tribler · 2023-04-17T05:29:20Z

Sentry issue: TRIBLER-15F

sentry-for-tribler · 2023-04-26T09:39:26Z

Sentry issue: TRIBLER-15Y

sentry-for-tribler · 2023-04-26T09:40:02Z

Sentry issue: TRIBLER-15X

sentry-for-tribler · 2023-04-26T09:48:25Z

Sentry issue: TRIBLER-15Q

sentry-for-tribler · 2023-11-06T15:47:22Z

Sentry issue: TRIBLER-1EF

sentry-for-tribler · 2023-11-23T10:11:39Z

Sentry issue: TRIBLER-1F3

sentry-for-tribler · 2023-11-23T10:12:38Z

Sentry issue: TRIBLER-1F2

sentry-for-tribler · 2023-11-27T11:02:17Z

Sentry issue: TRIBLER-1FE

kozlovsky · 2023-11-27T11:10:34Z

Fixed in #7628

devos50 added the type: bug label Mar 27, 2020

devos50 added this to the V7.5: core refactoring milestone Mar 27, 2020

ichorid modified the milestones: V7.5: core refactoring, Backlog Apr 1, 2020

ichorid modified the milestones: Backlog, V7.6: Faster, Quicker and more Responsive Jun 11, 2020

ichorid added the Trustchain label Jul 18, 2020

xoriole mentioned this issue Oct 22, 2020

Use SQLite to store torrent checkpoints data #5669

Closed

ichorid self-assigned this Nov 3, 2020

ichorid added the estimation: 3 days label Nov 3, 2020

drew2a modified the milestones: 7.6.0 November: Stability, usability, performance, Next-next release Nov 4, 2020

ichorid mentioned this issue Jan 12, 2021

[7.6.1] RuntimeError: TorrentState cannot be stored in the database #5928

Closed

drew2a modified the milestones: Next-next release, Backlog Sep 15, 2021

ichorid assigned kozlovsky and unassigned ichorid Sep 28, 2021

drew2a changed the title ~~[7.5.0-RC2] database disk image is malformed~~ Database disk image is malformed Nov 22, 2021

drew2a removed component: trustchain labels Dec 19, 2022

drew2a mentioned this issue Dec 19, 2022

Add ProcessLocker to ensure that no more than one Tribler GUI/Core process runs locally at any moment #7212

Merged

kozlovsky mentioned this issue Oct 16, 2023

Handle the "DatabaseError: database disk image is malformed" error #7628

Merged

kozlovsky modified the milestones: Backlog, 7.13.1 Nov 20, 2023

kozlovsky closed this as completed Nov 27, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Database disk image is malformed #5252

Database disk image is malformed #5252

devos50 commented Mar 27, 2020 •

edited by drew2a

Loading

qstokkink commented Mar 27, 2020

ichorid commented Mar 27, 2020

devos50 commented Apr 1, 2020

ichorid commented Apr 1, 2020

devos50 commented Jul 20, 2020

xoriole commented Oct 21, 2020

ichorid commented Oct 22, 2020

xoriole commented Oct 22, 2020

ghost commented Nov 3, 2020

ichorid commented Nov 3, 2020

devos50 commented Nov 3, 2020

sentry-for-tribler bot commented Nov 5, 2020

ichorid commented Nov 5, 2020

rgoshko commented Nov 6, 2020 •

edited by drew2a

Loading

ichorid commented Nov 6, 2020

rgoshko commented Nov 9, 2020

sentry-for-tribler bot commented Nov 20, 2020

sentry-for-tribler bot commented Dec 30, 2020

ichorid commented Oct 12, 2021

sentry-for-tribler bot commented Nov 22, 2021

sentry-for-tribler bot commented Dec 13, 2021

sentry-for-tribler bot commented Dec 24, 2021

sentry-for-tribler bot commented Mar 7, 2022 •

edited by drew2a

Loading

sentry-for-tribler bot commented Mar 7, 2022

sentry-for-tribler bot commented Dec 19, 2022

sentry-for-tribler bot commented Apr 17, 2023

sentry-for-tribler bot commented Apr 26, 2023

sentry-for-tribler bot commented Apr 26, 2023

sentry-for-tribler bot commented Apr 26, 2023

sentry-for-tribler bot commented Nov 6, 2023

sentry-for-tribler bot commented Nov 23, 2023

sentry-for-tribler bot commented Nov 23, 2023

sentry-for-tribler bot commented Nov 27, 2023

kozlovsky commented Nov 27, 2023

Database disk image is malformed #5252

Database disk image is malformed #5252

Comments

devos50 commented Mar 27, 2020 • edited by drew2a Loading

qstokkink commented Mar 27, 2020

ichorid commented Mar 27, 2020

devos50 commented Apr 1, 2020

ichorid commented Apr 1, 2020

devos50 commented Jul 20, 2020

xoriole commented Oct 21, 2020

ichorid commented Oct 22, 2020

xoriole commented Oct 22, 2020

ghost commented Nov 3, 2020

ichorid commented Nov 3, 2020

devos50 commented Nov 3, 2020

sentry-for-tribler bot commented Nov 5, 2020

ichorid commented Nov 5, 2020

rgoshko commented Nov 6, 2020 • edited by drew2a Loading

ichorid commented Nov 6, 2020

rgoshko commented Nov 9, 2020

sentry-for-tribler bot commented Nov 20, 2020

sentry-for-tribler bot commented Dec 30, 2020

ichorid commented Oct 12, 2021

sentry-for-tribler bot commented Nov 22, 2021

sentry-for-tribler bot commented Dec 13, 2021

sentry-for-tribler bot commented Dec 24, 2021

sentry-for-tribler bot commented Mar 7, 2022 • edited by drew2a Loading

sentry-for-tribler bot commented Mar 7, 2022

sentry-for-tribler bot commented Dec 19, 2022

sentry-for-tribler bot commented Apr 17, 2023

sentry-for-tribler bot commented Apr 26, 2023

sentry-for-tribler bot commented Apr 26, 2023

sentry-for-tribler bot commented Apr 26, 2023

sentry-for-tribler bot commented Nov 6, 2023

sentry-for-tribler bot commented Nov 23, 2023

sentry-for-tribler bot commented Nov 23, 2023

sentry-for-tribler bot commented Nov 27, 2023

kozlovsky commented Nov 27, 2023

devos50 commented Mar 27, 2020 •

edited by drew2a

Loading

rgoshko commented Nov 6, 2020 •

edited by drew2a

Loading

sentry-for-tribler bot commented Mar 7, 2022 •

edited by drew2a

Loading