Could not find element type modelling_realm_-_primaries in the vocabulary server.

[riazann]

Dear, developers!
I tried to start "HowTo-03a_Find_Experiments_and_Variables_for_given_Opportunities".

After running the line
DR = dr.DataRequest.from_separated_inputs(**content_dic)
I get an error:
"Could not find element type modelling_realm_-_primaries in the vocabulary server."

What does it mean? Could you help me, please.

[matthew-mizielinski]

Hi @riazann,

I've been able to reproduce the error you have found, but only by running on google colab. Running on my local systems doesn't hit this problem. I'll raise an issue to investigate this failure.

In the mean time note that there is an alternative example here that works fine on google colab for me.

[matthew-mizielinski]

#118 created

[rigoudyg]

@riazann
We have fixed the issue.
There was the issue when using the last version of the content but with a version of the software which is not up-to-date with this content.

[sol1105]

@rigoudyg This seems still an issue when using the latest software with old content versions 1.0 and 1.1. From content v1.2 it works.

[rigoudyg]

@sol1105 Thank you for pointing me this issue. I do not manage to have the same issue as you have. Can you giv me the commit number of the software you used for this?
Best

[sol1105]

@rigoudyg
It happens for v1.0 and v1.1 with the latest commit with default configuration. If I set consolidation = False it works, however. So, to reproduce the error you need set consolidation = True and remove any previously generated VS/DR files from the cache.

The issue seems to be that in the consolidation I set the Variables field Modelling Realm to Modelling Realm - Primary, to be consistent between versions <v1.2 that call the field Modelling Realm and >=v1.2 which call the field Modelling Realm - Primary. Your logic in dump_transformation.py does not expect that vor <v1.2 and throws an error for DR = dr.DataRequest.from_separated_inputs(**content_dic) with the following traceback.

File "CMIP7_DReq_Software/data_request_api/data_request_api/query/data_request.py", line 569, in from_separated_inputs
    VS = VocabularyServer.from_input(VS_input)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 80, in from_input
    return cls(content)
           ^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 70, in __init__
    self.check_infinite_loop()
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 136, in check_infinite_loop
    found = any(follow_loop(key) for key in sorted(list(call_dict)))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 136, in <genexpr>
    found = any(follow_loop(key) for key in sorted(list(call_dict)))
                ^^^^^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 132, in follow_loop
    found = found or follow_loop(next_key, former_keys + [alias_key, ])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 126, in follow_loop
    alias_key, _ = self.get_element_type_ids(current_key)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 159, in get_element_type_ids
    element_type = self.get_element_type(element_type)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "CMIP7_DReq_Software/data_request_api/data_request_api/query/vocabulary_server.py", line 151, in get_element_type
    raise ValueError(f"Could not find element type {element_type} in the vocabulary server.")
ValueError: Could not find element type modelling_realm_-_primaries in the vocabulary server.

I think the issue comes from modelling_realm.+ not being merged to modeling_realm for <1.2. If I change the following it works:

--- a/data_request_api/data_request_api/content/transform.json
+++ b/data_request_api/data_request_api/content/transform.json
@@ -167,8 +167,20 @@
         }
       }
     },
-    "v1.0": {},
-    "v1.1": {},
+    "v1.0": {
+      "keys_to_merge": {
+        "variables": {
+          "modelling_realm.+": "modelling_realm"
+         }
+      }
+    },
+    "v1.1": {
+      "keys_to_merge": {
+        "variables": {
+          "modelling_realm.+": "modelling_realm"
+        }
+      }
+    },

Do you think this change will cause any other issues? I did test it with both, consolidation = True / False and did not get the error anymore.

Generally, maybe the release export should have the fieldname Modelling Realm - Primary set to Modelling Realm in coming versions as that would make the consolidation easier? If not, I think, the consolidation should keep changing Modelling Realm to Modelling Realm - Primary for <v1.2 to have a consistent field name across versions.