Directory: Fix directory migration crash

[Tschuppi81]

Directory: Fix directory migration crash when renaming option labels

TYPE: Bugfix
LINK: ogc-2353

[linear]

OGC-2353 Crash in Verzeichnis-Migration wenn eine existierende Auswahl entfernt wird

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
2607	1	2606	17

View the top 1 failed test(s) by shortest run time

tests/onegov/directory/test_orm.py::test_introduce_required_field_fail

Stack Traces | 1.42s run time

session = <sqlalchemy.orm.session.Session object at 0x7fb1e9e91dd0>

    def test_introduce_required_field_fail(session: Session) -> None:
        rooms = DirectoryCollection(session).add(
            title="Rooms",
            structure="""
                Name *= ___
            """,
            configuration=DirectoryConfiguration(
                title='Name',
                order=['Name'],
            )
        )
    
        rooms.add(values=dict(
            name="Conference Room",
        ))
    
        rooms.structure = """
            Name *= ___
            Seats *= 0..99
        """
    
        with pytest.raises(ValidationError):
>           session.flush()

.../onegov/directory/test_orm.py:583: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
............/app/lib/python3.11.../sqlalchemy/orm/session.py:2540: in flush
    self._flush(objects)
............/app/lib/python3.11.../sqlalchemy/orm/session.py:2569: in _flush
    self.dispatch.before_flush(self, flush_context, objects)
............/app/lib/python3.11.../sqlalchemy/event/attr.py:261: in __call__
    fn(*args, **kw)
............/app/lib/python3.11.../site-packages/sqlalchemy_utils/observer.py:321: in invoke_callbacks
    callback(root_obj, *[objs[i] for i in range(len(objs))])
.../core/orm/observer.py:148: in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
.../directory/models/directory.py:477: in structure_configuration_observer
    self.migration(structure, configuration).execute()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <onegov.directory.migration.DirectoryMigration object at 0x7fb1e9df4b90>

    def execute(self) -> None:
        """ To run the migration, run this method. The other methods below
        should only be used if you know what you are doing.
    
        """
>       assert self.possible
               ^^^^^^^^^^^^^
E       AssertionError

.../onegov/directory/migration.py:115: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

[Daverball]

This looks like a decent start, but we need to allow more operations than renaming a single option, there's many other legal changes, that will not result in form validation errors on existing entries.

[Daverball]

Looks pretty good. We're getting close, but I think we're now being a bit to laissez-faire with renaming options.

[Daverball]

It seems slightly controversial to delete existing selections. We might want to add an additional warning/confirmation step, if we detect that removing/renaming an option would modify the recorded data of existing entries (it would always be fine if none of the existing entries have selected this option).

[Tschuppi81]

It is actually the same as with removing fields.
This is what you currently need to confirm:

Is that what you meant (A) or do you think of an extra step (B) where each entry would need to be changed in a separate window before an option can be removed?

[Daverball]

Yes, although it might be nice to be a bit more explicit that some data will be lost if there are existing entries that use the removed option.

[Daverball]

Are we doing a good enough job of visualizing the ValidationError that can be emitted by Directory.update as part of DirectoryMigration.execute? E.g. when we add a required field to a directory with existing entries, are we getting a better error than "Eingabe erforderlich", do we see which field caused the issue? It might be worth detecting this common mistake, to provide a better error message, since you can never add a required field (other than a checkbox/radio field with a pre-selected option) to a directory that already has some entries, you will always need to make it optional first, fill in the field for all existing entries and then update the field to be required.

[Tschuppi81]

The ValidationError pops up saying which field is required. But when opening the link to the entry the new field does not exist yet so we end up in a deadlock.
I will detect this mistake and through an error to the user preventing required fields in one step