Update shared specification with new notation

templates/attachment_extraction_acceptance_test.plain

@@ -1,10 +1,10 @@
-Test The Extraction Function with the following flow (The Attachment Test Flow):
-  - Step 1: Invoke The Extraction Function using the resource [{{ data_extraction_resource_name }}](test_data/{{ data_extraction_resource_name }}).
-    - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
-  - Step 2: Invoke The Extraction Function using the resource [{{ attachment_extraction_resource_name }}](test_data/{{ attachment_extraction_resource_name }}).
-    - Expect The Callback Server to receive from DevRev a **single** event (`:= event_from_callback_server`) with "event_type" that equals "EXTRACTION_ATTACHMENTS_DONE".
+Test :ExtractionFunction: with the following flow (`:= attachmentTestFlow`):
+  - Step 1: Invoke :ExtractionFunction: using the resource [{{ data_extraction_resource_name }}](test_data/{{ data_extraction_resource_name }}).
+    - Expect :TestCallbackServer: to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
+  - Step 2: Invoke :ExtractionFunction: using the resource [{{ attachment_extraction_resource_name }}](test_data/{{ attachment_extraction_resource_name }}).
+    - Expect :TestCallbackServer: to receive from DevRev a **single** event (`:= event_from_callback_server`) with "event_type" that equals "EXTRACTION_ATTACHMENTS_DONE".
     - Expect `event_from_callback_server["event_data"]["artifacts"]` to be an array. Expect this array to not be empty. Expect this array to have length 1.
     - Let `artifact_object = event_from_callback_server["event_data"]["artifacts"][0]`.
       - Expect `artifact_object["item_type"] = "ssor_attachment"` and `artifact_object["item_count"] = {{ expected_attachment_count }}`.
       - Make a GET request to "http://localhost:8003/is_uploaded/{artifact_object['id']}". Expect response code to be equal to 200. 
-  - Note: The Attachment Test Flow should be executed within a single test.
+  - Note: attachmentTestFlow should be executed within a single test.

templates/chef_cli_normalization_validation.plain

@@ -1,10 +1,14 @@
-validate The Normalization Function with the chef-cli tool (The Chef CLI) using the command "chef-cli validate-data -m <The External Domain Metadata JSON object file> -r {{ record_type }}":
-  - To invoke the function that returns The External Domain Metadata JSON object, send the event [external_domain_metadata_event_payload.json](test_data/external_domain_metadata_event_payload.json) to The Snap-In server.
-  - The External Domain Metadata JSON object should then be stored in temporary file and provided as path to the validation of The Normalization Function.
-  - The Chef CLI reads the extracted file (The Extracted File) from stdin.
-  - The Extracted File is inside folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH". To retrieve the relevant file name, command `ls EXTRACTED_FILES_FOLDER_PATH | grep extractor_{{ record_type }} | sort -r | head -n 1` should be executed. The test should always fail if The Extracted File is not present.
-  - Please note that the expected behaviour is that only after the invocation of The Extraction Function, the folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH" should exist. The test should always fail if before the execution of The Chef CLI, the folder "EXTRACTED_FILES_FOLDER_PATH" does not exist or if The Extracted File is not found.
-  - The path to The Chef CLI executable is provided in the environment variable `CHEF_CLI_PATH`.
-  - For the validation of The Normalization Function to be successful The Chef CLI must return an empty output. The implementation of The Extraction Function directly infuences the output of The Chef CLI.
-  - The stdout and stderr of The Chef CLI *must* be printed to the console.
-  - The test should always fail if The Chef CLI is not available.
+Validate :NormalizationFunction: with :ChefCLI: for the "{{ record_type }}" record type with the following flow:
+  - Step 1.: Invoke :ExtractionFunction: using the resource [data_extraction_test.json](test_data/data_extraction_test.json).
+  - Step 2.: Retrieve full extracted file path (`:= fullExtractedFilePath`)
+    - The extracted file name (`:= extractedFileName`) is the name of the file created by :ExtractionFunction:
+    - extractedFileName is located in the folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH".
+    - To retrieve extractedFileName, execute the following command: `ls EXTRACTED_FILES_FOLDER_PATH | grep extractor_{{ record_type }} | sort -r | head -n 1`
+    - The test should fail the folder "EXTRACTED_FILES_FOLDER_PATH" does not exist or if the extractedFileName is not found.
+    - fullExtractedFilePath is the full path to the extractedFileName.
+  - Step 3.: In :ConformanceTests:, create a separate file for :ExternalDomainMetadataJSONObject:. Contents of :ExternalDomainMetadataJSONObject: JSON object **MUST NOT** be modified compared to the original file contents.
+  - Step 4.: Validate :NormalizationFunction: with :ChefCLI: using the command `"${CHEF_CLI_PATH}" validate-data -m <path to :ExternalDomainMetadataJSONObject: file> -r {{ record_type }} < <fullExtractedFilePath>`:
+    - Validation is successful precisely when both the stdout and stderr of :ChefCLI: are empty.
+    - Expect the validation of :ChefCLI: to succeed. If it fails, normalization method for "{{ record_type }}" in :Implementation: should be fixed.
+    - For the validation of :NormalizationFunction: to be successful The :ChefCLI: must return an empty output. Note that the implementation of :ExtractionFunction: directly infuences the output of :ChefCLI:.
+    - The stdout and stderr of :ChefCLI: *MUST* be printed to the console.

templates/data_extraction_acceptance_test.plain → templates/data_extraction_acceptance_test_no_pagination.plain

@@ -1,5 +1,5 @@
-Test The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}).
-  - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
+Test :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}).
+  - Expect :TestCallbackServer: to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
   - Let `artifact_array = <event_from_callback_server>["event_data"]["artifacts"]` (array of objects).
     - Expect `len(artifact_array) > 0`.
     - Out of `artifact_array`, expect one of the elements to have "item_type" equal to "{{ artifact_name }}" (`:= {{ artifact_name }}_artifact`) and `{{ artifact_name }}_artifact["item_count"]={{ expected_item_count }}`.

templates/data_extraction_acceptance_test_with_pagination.plain

@@ -0,0 +1,7 @@
+Test :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}).
+  - Expect :TestCallbackServer: to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE".
+  - Let `artifact_array = <event_from_callback_server>["event_data"]["artifacts"]` (array of objects).
+    - Expect `len(artifact_array) > 0`.
+    - Out of `artifact_array`, expect one of the elements to have "item_type" equal to "{{ artifact_name }}" (`:= {{ artifact_name }}_artifact`) and `{{ artifact_name }}_artifact["item_count"]={{ expected_item_count }}`.
+    - If `{{ artifact_name }}_artifact["item_count"] < {{ expected_item_count }}`, this indicates that not all the {{ artifact_name }} data was extracted.
+    - If `{{ artifact_name }}_artifact["item_count"] < {{ expected_item_count }}` and equals {{ fetched_page_size }}, it means :ExtractionFunction: fetched only one page instead of all pages. Pages should be fetched iteratively until all pages are fetched.

templates/data_fetching_function_resource.plain

@@ -0,0 +1 @@
+Implement the :Function: "fetch_{{ record_type }}" that fetches all {{ record_type }} as specified in :ObjectPRD: [{{ record_type }}-extraction.md](product_requirements/{{ record_type }}-extraction.md) using {{ pagination_rule }}.

templates/data_fetching_return_expectation.plain

@@ -1,5 +1,5 @@
-The Functions implemented *should* also return:
-  - `status_code` - a status code from The API.
-  - `api_delay` - Delay in seconds to wait before retrying the request to The API (if no rate limit, set to 0)
-  - `message` - Information about the call the The Function.
-  - Note: Every The Function should handle rate limiting.
+:Function: implemented *should* also return:
+  - `status_code` - a status code from :API:.
+  - `api_delay` - Delay in seconds to wait before retrying the request to :API: (if no rate limit, set to 0)
+  - `message` - Information about the call :Function:.
+  - `data` (optional) - Data returned from :API:.

templates/edm_record_type_template.plain

@@ -0,0 +1 @@
+The :Function: 'get_external_domain_metadata' should return :ExternalDomainMetadataJSONObject: with the record type "{{ record_type }}" based on external fields from file [{{ product_requirement_file }}]({{ product_requirement_file }}), while preserving any existing record types.

templates/external_domain_metadata_boilerplate.plain

@@ -1,30 +1,36 @@
 ***Definitions:***
 
-- The structure of The External Domain Metadata JSON object is specified by the JSON schema defined in the resource [external_domain_metadata_schema.json](external_domain_metadata_schema.json).
+- The structure of :ExternalDomainMetadataJSONObject: is specified by the JSON schema defined in the resource [external_domain_metadata_schema.json](external_domain_metadata_schema.json).
   - Please note that all refers_to keys in reference and typed_reference fields must use the format "#record:<record_type_key>", where <record_type_key> matches a key in the top-level record_types. This ensures references are unambiguous and valid. Use {} as the value to refer by default identifier (id), or specify { "by_field": "field_name" } to use a custom key.
 
 - A “display name” is the human-readable label of a field within a given record type.
 
 ***Non-Functional Requirements:***
 
-- Store The External Domain Metadata JSON object as a separate JSON file.
+- Store :ExternalDomainMetadataJSONObject: as a separate JSON file.
 
-- The External Domain Metadata JSON object *MUST* be imported without "as" statement in The Implementation Code.
+- :ExternalDomainMetadataJSONObject: *MUST* be imported without "as" statement in :Implementation:.
 
-- {% include "templates/should_not_modify_extraction_function.plain" %}
+- The :UnitTests: *MUST NOT* test the contents and *MUST NOT* teset the  of :ExternalDomainMetadataJSONObject:.
 
 ***Test Requirements:***
 
-- Validate generated The External Domain Metadata JSON object with the chef-cli tool (The Chef CLI) using the command "chef-cli validate-metadata":
+- Validate generated :ExternalDomainMetadataJSONObject: with :ChefCLI: using the command "${CHEF_CLI_PATH} validate-metadata":
   - This command reads JSON object from stdin.
-  - The path to The Chef CLI executable is provided in the environment variables `CHEF_CLI_PATH`.
-  - For the validation of The External Domain Metadata JSON object to be successful The Chef CLI must return an empty output.
-  - The test should always fail if The Chef CLI is not available.
+  - The path to :ChefCLI: executable is provided in the environment variables `CHEF_CLI_PATH`.
+  - For the validation of :ExternalDomainMetadataJSONObject: to be successful :ChefCLI: must return an empty output.
+  - The test should always fail if :ChefCLI: is not available.
 
-- The External Domain Metadata JSON object should always be read from The Implementation Code.
+- :ExternalDomainMetadataJSONObject: *MUST* be validated solely against :ChefCLI:.
+  - You *MUST NOT* validate :ExternalDomainMetadataJSONObject: against the literal JSON contents of :ExternalDomainMetadataJSONObject:.
+  - :ConformanceTests: should have *ONLY TWO* tests
+    - That :Function: returning :ExternalDomainMetadataJSONObject: can be invoked successfully
+    - :ChefCLI: validation using the command "${CHEF_CLI_PATH} validate-metadata"
 
-- If The Conformance Tests test The External Domain Metadata JSON object, test it only against The Chef CLI. Don't test it against the literal JSON contents of The External Domain Metadata JSON object.
+- :ExternalDomainMetadataJSONObject: should always be read from :Implementation: (and not recreated in :ConformanceTests:)
 
-- The stdout and stderr of The Chef CLI *must* be printed to the console.
+- The stdout and stderr of :ChefCLI: *must* be printed to the console.
 
-- The Conformance Tests should not modify nor do any transformations on The External Domain Metadata JSON object at all. If The Conformance Tests indicates a problem with The External Domain Metadata JSON object, The External Domain Metadata JSON object in The Implementation Code should be adjusted to fix the problem. The tests should only validate the JSON object in its original form.
+- :ConformanceTests: should not modify nor do any transformations on :ExternalDomainMetadataJSONObject: at all.
+  - If :ConformanceTests: indicates a problem with :ExternalDomainMetadataJSONObject:, :ExternalDomainMetadataJSONObject: in :Implementation: should be adjusted to fix the problem.
+  - :ConformanceTests: should only validate the JSON object in its original form.

templates/external_sync_unit_acceptance_test.plain

@@ -1,5 +1,5 @@
-Test The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). 
-  - Expect The Callback Server to receive *a single* event with "event_type" "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE".
+Test :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}). 
+  - Expect :TestCallbackServer: to receive *a single* event with "event_type" "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE".
   - Let `external_sync_units_array = <event_from_callback_server>["event_data"]["external_sync_units"]` (array of objects).
     - Expect `external_sync_units_array` to exist and be an array.
     - Expect `len(external_sync_units_array) = {{ expected_external_sync_unit_count }}`.

templates/extraction_function_resource.plain

@@ -0,0 +1,6 @@
+If "event_type" equals "EXTRACTION_DATA_START" or "EXTRACTION_DATA_CONTINUE" :ExtractionFunction: should extract the '{{ record_type }}' data, while preserving previous artifact pushes of data.
+  - Refer to resource [{{ record_type }}-extraction.md](product_requirements/{{ record_type }}-extraction.md) for details on {{ record_type }} extraction specification.
+  - Refer to resource [high-level-data-extraction-plan.md](product_requirements/high-level-data-extraction-plan.md) for:
+    - The order of artifact extraction.
+    - Information if we should use pagination or not for the extraction of the '{{ record_type }}' data.
+  - {{additional_instructions}}