InotifyTree does not pick up file events if the directories created

[oonisim]

Environment

• inotify 0.2.10
• Ubuntu 22.04 LTS
• Python 3.10.12

Reproducing the issue

If create the directories after monitoring started, file events e.g CLOSE_WAIT do get picked up.

import os
import pathlib
from inotify.adapters import (
    InotifyTree
)
from inotify.constants import (
    IN_ALL_EVENTS,
)


def test():
    with tempfile.TemporaryDirectory() as directory:
        monitor: InotifyTree = InotifyTree(path=directory, mask=IN_ALL_EVENTS)
        events = monitor.event_gen(yield_nones=False)
        for file in [
            "20231112/0001.txt",
            "20231113/0002.txt",
        ]:
            path_to_file: str = os.path.join(directory, file)
            folder: str = str(pathlib.Path(path_to_file).parent)
            mkdir(folder)
            with open(file=path_to_file, mode="w", encoding='utf-8') as _file:
                _file.write("foobar")

        while True:
            _, event_names, folder, filename = next(events)
            print(f"{folder} {filename} {event_names}")
---
-------------------------------- live log call ---------------------------------
DEBUG    inotify.adapters:adapters.py:58 Inotify handle is (5).
DEBUG    inotify.adapters:adapters.py:343 Adding initial watches on tree: [/tmp/tmphtf2h_ts]
DEBUG    inotify.adapters:adapters.py:82 Adding watch: [/tmp/tmphtf2h_ts]
DEBUG    inotify.adapters:adapters.py:96 Added watch (1): [/tmp/tmphtf2h_ts]
DEBUG    inotify.adapters:adapters.py:228 Events received from epoll: ['IN_ACCESS']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_CREATE', 'IN_ISDIR']
DEBUG    inotify.adapters:adapters.py:295 A directory has been created. We're adding a watch on it (because we're being recursive): [/tmp/tmphtf2h_ts/20231112]
DEBUG    inotify.adapters:adapters.py:82 Adding watch: [/tmp/tmphtf2h_ts/20231112]
DEBUG    inotify.adapters:adapters.py:96 Added watch (2): [/tmp/tmphtf2h_ts/20231112]
/tmp/tmphtf2h_ts 20231112 ['IN_CREATE', 'IN_ISDIR']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_CREATE', 'IN_ISDIR']
DEBUG    inotify.adapters:adapters.py:295 A directory has been created. We're adding a watch on it (because we're being recursive): [/tmp/tmphtf2h_ts/20231113]
DEBUG    inotify.adapters:adapters.py:82 Adding watch: [/tmp/tmphtf2h_ts/20231113]
DEBUG    inotify.adapters:adapters.py:96 Added watch (3): [/tmp/tmphtf2h_ts/20231113]
/tmp/tmphtf2h_ts 20231113 ['IN_CREATE', 'IN_ISDIR']

If the directories created in-advance, file events e.g CLOSE_WAIT get picked up.

def test2():
    with tempfile.TemporaryDirectory() as directory:
        mkdir(os.path.join(directory, "20231112"))
        mkdir(os.path.join(directory, "20231113"))

        monitor: InotifyTree = InotifyTree(path=directory, mask=IN_ALL_EVENTS)
        events = monitor.event_gen(yield_nones=False)
        for file in [
            "20231112/0001.txt",
            "20231113/0002.txt",
        ]:
            path_to_file: str = os.path.join(directory, file)
            folder: str = str(pathlib.Path(path_to_file).parent)
            mkdir(folder)
            with open(file=path_to_file, mode="w", encoding='utf-8') as _file:
                _file.write("foobar")

        while True:
            _, event_names, folder, filename = next(events)
            print(f"{folder} {filename} {event_names}")

-------------------------------- live log call ---------------------------------
DEBUG    inotify.adapters:adapters.py:58 Inotify handle is (5).
DEBUG    inotify.adapters:adapters.py:343 Adding initial watches on tree: [/tmp/tmpdlzodbg_]
DEBUG    inotify.adapters:adapters.py:82 Adding watch: [/tmp/tmpdlzodbg_]
DEBUG    inotify.adapters:adapters.py:96 Added watch (1): [/tmp/tmpdlzodbg_]
DEBUG    inotify.adapters:adapters.py:82 Adding watch: [/tmp/tmpdlzodbg_/20231112]
DEBUG    inotify.adapters:adapters.py:96 Added watch (2): [/tmp/tmpdlzodbg_/20231112]
DEBUG    inotify.adapters:adapters.py:82 Adding watch: [/tmp/tmpdlzodbg_/20231113]
DEBUG    inotify.adapters:adapters.py:96 Added watch (3): [/tmp/tmpdlzodbg_/20231113]
DEBUG    inotify.adapters:adapters.py:228 Events received from epoll: ['IN_ACCESS']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_CREATE']
/tmp/tmpdlzodbg_/20231112 0001.txt ['IN_CREATE']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_OPEN']
/tmp/tmpdlzodbg_/20231112 0001.txt ['IN_OPEN']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_MODIFY']
/tmp/tmpdlzodbg_/20231112 0001.txt ['IN_MODIFY']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_CLOSE_WRITE']
/tmp/tmpdlzodbg_/20231112 0001.txt ['IN_CLOSE_WRITE']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_CREATE']
/tmp/tmpdlzodbg_/20231113 0002.txt ['IN_CREATE']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_OPEN']
/tmp/tmpdlzodbg_/20231113 0002.txt ['IN_OPEN']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_MODIFY']
/tmp/tmpdlzodbg_/20231113 0002.txt ['IN_MODIFY']
DEBUG    inotify.adapters:adapters.py:169 Events received in stream: ['IN_CLOSE_WRITE']
/tmp/tmpdlzodbg_/20231113 0002.txt ['IN_CLOSE_WRITE']

[oonisim]

This can be expected as https://github.com/dsoprea/PyInotify#notes. The sample code is creating the directory in the same single thread/CPU, then InotifyTree can miss the timing.

IMPORTANT:
Recursively monitoring paths is not a functionality provided by the kernel.
Rather, we artificially implement it. As directory-created events are received,
we create watches for the child directories on-the-fly.

This means that there is potential for a race condition:
if a directory is created and a file or directory is created inside before you
(using the event_gen() loop) have a chance to observe it, then you are going to
have a problem: If it is a file, then you will miss the events related to its
creation, but, if it is a directory, then not only will you miss those creation
events but this library will also miss them and not be able to add a watch for them.

[rpl-ian-lunam]

This is also a problem if the directory is created with a mkdir -p which also create parent directories. Only the first new parent in the tree is noticed.

ie: if /a/b is being watched and mkdir -p /a/b/c/d/e is run only the contents of /a/b/c will be added to the watch. Any subsequent events in /a/b/c/d or /a/b/c/d/e will not be captured.

This should be an easy fix. When a dir is added, check if it has subdirs and add them.

[rpl-ian-lunam]

I'm not good enough with python to provide a PR, but it seems to me that the add-watch should be calling the __load_trees but that's in another class.

[ulidtko]

At least with system software, it is instructive to descend "one level below", to the roots.

inotifywait is a tool in the inotify-tools package.

Its manpage (man 1 inotifywait) has an enlightening section BUGS which says:

There are race conditions in the recursive directory watching code
which can cause events to be missed if they occur in a directory
immediately after that directory is created. This is probably not
fixable.

And that's damn right!

Because there's no atomic operation to add a watch recursively over a tree at once -- i.e. this must be made in multiple calls to inotify API -- it's then unavoidable that the target tree might change, right below our feet, right at the time while we're doing the initial walk, discovering all the subdirs there, and adding individual watches over those. Race conditions like this, and similar, are bound to happen. It's a limitation of the API.

And of course, python's slowness exacerbates the issue. No I agree, it can't be fixed.