in_tail: fluent-bit reads wrong offsets when two file have the same name and the same inode on linux system.

[wtan825]

Bug Report

Describe the bug
when i read the code, i find that fluent-bit use file name and inode to set the checkpoints in db.(https://github.com/fluent/fluent-bit/blob/master/plugins/in_tail/tail_db.c line 109). the problem is after file (named A) is deleted, another file ( also named A) created with the same inode. fluent-bit will read the old A's offset.

int flb_tail_db_file_set(struct flb_tail_file *file,
                         struct flb_tail_config *ctx)
{
    int ret;
    char query[PATH_MAX];
    struct query_status qs = {0};
    uint64_t created;

    /* Check if the file exists */
    snprintf(query, sizeof(query) - 1,
             SQL_GET_FILE,
             file->name, file->inode);

    memset(&qs, '\0', sizeof(qs));
    ret = flb_sqldb_query(ctx->db,
                          query, cb_file_check, &qs);

To Reproduce

dd if=/dev/zero of=disk.img bs=1M count=128
mkfs.ext4 disk.img
mkdir mnt
mount -o loop disk.img mnt
cd mnt
echo "hello" >a.log
ls -li 
rm -f a.log
echo "again" >a.log
ls -li

in mnt, delete a.log, then create a.log. the two files may have the same inode. besides, two files which have the same name in different device may have the same inode.

Expected behavior
when two files have the same name and the same inode in different time would not affect each other

Your Environment
linux

[edsiper]

@wtan825

thanks for reporting this problem and providing steps to reproduce it!

I've pushed fix f329345 which addresses the problem.

[wtan825]

@wtan825

thanks for reporting this problem and providing steps to reproduce it!

I've pushed fix f329345 which addresses the problem.

@edsiper
thanks for addressing the problem. I have read the fix f329345, but this will bring up another problem： if we want to stop monitoring a inactive file, we will delete file offset in db. when the file turns to be active again, we cannot find the right offset. although fluent-bit does not have this function for the moment，stopping monitoring a inactive file is necessary in some circumstances.

[edsiper]

@wtan825 I don't understand the last problem described. We stop monitoring a file only if the file gets deleted or rotated.

[wtan825]

e only if the file gets deleted or rotated.

@edsiper Now We just stop monitoring a file only if the file gets deleted or rotated. But in some circumstances we hope to stop monitoring a inactive file. For instance, when we monitoring a directory, the path is set to /var/applog/*. If we do not stop monitoring a inactive file, too many files will be monitored. filebeat has this feature which could stop monitoring a inactive file.

[edsiper]

@wtan825 what would be the expected solution from a configuration perspective ? something like: inactive_timeout = 30; where after 30 seconds if the file don't get any new data just stop monitoring it and discard it ? (even if after a minute it gets more data?)

[wtan825]

@wtan825 what would be the expected solution from a configuration perspective ? something like: inactive_timeout = 30; where after 30 seconds if the file don't get any new data just stop monitoring it and discard it ? (even if after a minute it gets more data?)

@edsiper yes. for instance, inactive_timeout = 1d. after 1d if the file don't get any new data, we just stop monitoring it. but still keep offset. offset will be deleted only after the file is deleted.

[srini38]

@edsiper Can this fix be applied to fluent-bit 1.2.2 ?

[edsiper]

@srini38 nop, all fixes are being applied now on the v1.4 series.

[srini38]

@edsiper Sorry I should have been clearer. Can I apply this to my copy of fluent-bit 1.2.2 code. Do you see any issues?

[edsiper]

hmm not 100% sure, but you will have to try.

why not upgrade to the latest v1.4.2 ?

[srini38]

@edsiper Will try it out. Unfortunately cannot upgrade to 1.4.x now.

[edsiper]

We have implemented other fixes to detect files rotation, please use v1.4.6 to get the solution in place.

ref: https://fluentbit.io/announcements/v1.4.6/

closing as fixed.

[ganga1980]

@edsiper, @patrick-stephens , what's the latest version which has the fix for this issue? does this fix work for upgrade scenario?
As reported here - #4895 and I am still seeing this issue. can you please help on triaging this issue - #4895 ?

[ganga1980]

@edsiper, @patrick-stephens , can you guys help on triaging this issue - #4895 as we have couple of customers waiting for the fix?

[patrick-stephens]

It looks like there is an open issue for it so will leave it with that. They may be related, I'm not sure and not really the expert so will leave others to comment there.

[ganga1980]

It looks like there is an open issue for it so will leave it with that. They may be related, I'm not sure and not really the expert so will leave others to comment there.

@patrick-stephens , thanks for the response. There is already open issue #4895 which created back in Feb. Not sure, whether I need to create new issue to get that triaged. let me know what would be best way to get this issue triaged.

[hsingli20]

it looks even two files with different file names, when the old file is deleted, but the stale data in sqllite db will lead the offset issue. Then it will cause the log file output truncated.