[History] Same task not being appended

[tashrifbillah]

Description

Same task is being overwritten in the task history as opposed to being appended.

Way to reproduce

luigi.cfg

[core]
default-scheduler-url = http://localhost:8082

[scheduler]
record_task_history = True
state_path = /tmp/luigi-state.pickle

[task_history]
db_connection = sqlite:////tmp/luigi-task-hist.db

[worker]
check_complete_on_run = True

From the same directory of luigi.cfg, run this gist.

• run once

python sample_luigi_task.py

• clean up target files

rm /tmp/?.txt

• run again

python sample_luigi_task.py

Actual behavior

Run this gist to view contents of .db file, history can be also viewed in /history tab of the web visualizer

python sqlite_db_browser /tmp/luigi-task-hist.db

(1, 'TaskD__99914b932b', 'TaskD', 'pnl_servers')
(2, 'TaskB__99914b932b', 'TaskB', 'pnl_servers')
(3, 'TaskA__99914b932b', 'TaskA', 'pnl_servers')
(4, 'TaskC__99914b932b', 'TaskC', 'pnl_servers')

Expected behavior

The above block should appear twice:

(1, 'TaskD__99914b932b', 'TaskD', 'pnl_servers')
(2, 'TaskB__99914b932b', 'TaskB', 'pnl_servers')
(3, 'TaskA__99914b932b', 'TaskA', 'pnl_servers')
(4, 'TaskC__99914b932b', 'TaskC', 'pnl_servers')
(5, 'TaskD__99914b932b', 'TaskD', 'pnl_servers')
(6, 'TaskB__99914b932b', 'TaskB', 'pnl_servers')
(7, 'TaskA__99914b932b', 'TaskA', 'pnl_servers')
(8, 'TaskC__99914b932b', 'TaskC', 'pnl_servers')

Environment

CentOS 7 Linux 3.10.0-1127.13.1.el7.x86_64
Python 3.7.6
sqlalchemy.__version__
'1.3.13'

sqlite3.version
'2.6.0'

luigi.__version__
'3.0.1'

Did anyone else experience this issue? It could be a bug.

[tashrifbillah]

Same task is being overwritten

Okay, this is what I observed: until a certain time, it is being overwritten and then appended. Can anyone tell which configuration parameter is controlling that?

I observed the certain time to be >3 hours.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If closed, you may revisit when your time allows and reopen! Thank you for your contributions.