recursive value expansion in PathExAttMap

[stolarczyk]

Is it intentional that the value expansion is performed only on the top level in attmap.PathExAttMap.__getitem__ method?

This makes the result of the literal value retrieval (lines 3 and 5) different from the same value retrieval (lines 4 and 6) in all the methods that rely on __getitem__, like items, to_dict etc.

The illustration uses a pipestat configuration file with environment variables read by yacman.YacAttMap, which inherits from attmap.PathExAttMap:

In [1]: from yacman import YacAttMap                                                                                                                                                                      

In [2]: y=YacAttMap(filepath="/Users/mstolarczyk/Desktop/testing/pypiper/pipestat_config.yaml")                                                                                                           

In [3]: y.database.port                                                                                                                                                                                   
Out[3]: '5432'

In [4]: y.to_dict(expand=True)["database"]["port"]                                                                                                                                                        
Out[4]: '$PRT'

In [5]: y.port                                                                                                                                                                                            
Out[5]: '5432'

In [6]: y.to_dict(expand=True)["port"]                                                                                                                                                                    
Out[6]: '5432'

In [7]: cat /Users/mstolarczyk/Desktop/testing/pypiper/pipestat_config.yaml                                                                                                                               
name: test
record_identifier: sample1
schema_path: sample_output_schema.yaml
port: $PRT
database:
  name: pipestat-test
  user: postgres
  password: pipestat-password
  host: localhost
  port: $PRT

---

A possible solution to this would be to change the current _safely_expand function from

def _safely_expand(x):
    return expandpath(x) if isinstance(x, str) else x

to

def _safely_expand(x):
     if isinstance(x, Mapping):
         return {k: _safely_expand(v) for k, v in x.items()}
    return expandpath(x) if isinstance(x, str) else x

I'm wondering if this would cause some serious downstream issues as most of our Python packages import this one.

[vreuter]

Sorry, meant to hit "cancel," not "close"

[vreuter]

That's a good idea, it's just that it would break the nested attribute access since _safely_expand is used in a few places.

[vreuter]

OK @stolarczyk I opened #75 to try to address this. It looks like it behaves on the test case you provided (though I had to export PRT=5432?) to test. FYI though, I have no idea what #75 may or may not break

[nsheff]

I believe it may have been intentional that the to_dict methods would not change anything, as they would then provide a way to retrieve the original, non-expanded values.

If we change this behavior, will there be another way to get the non-expanded values?

I don't remember the use case and maybe I'm wrong about the original intention, but I have a vague memory of this.

[stolarczyk]

Yes, there is a way to get the original values. You just omit the 'expand=True' in the method calls.