diff --git a/devrev-snapin-template.plain b/devrev-snapin-template.plain index 7da474a..0d933d9 100644 --- a/devrev-snapin-template.plain +++ b/devrev-snapin-template.plain @@ -1,128 +1,181 @@ ***Definitions:*** -- The Snap-In is a TypeScript Node.js application. +- :SnapIn: is a TypeScript Node.js application. -- The Function implements part of The Snap-In functionality. - - Details on how The Function must be invoked are provided in the resource [Function Invocation](docs/function_invocation.mdx). - - The Function should only take the first event from the array of events and process it. The Function can be invoked multiple times for multiple events. +- :Implementation: is how :SnapIn: is implemented. -- The Airdrop SDK Library for TypeScript helps developers build snap-ins that integrate with DevRev’s Airdrop platform. - - The documentation for the The Airdrop SDK Library is provided in the resource [airdrop_sdk_library_documentation.md](docs/airdrop_sdk_library_documentation.md). +- :Function: implements part of :SnapIn: functionality. + - Details on how :Function: must be invoked are provided in the resource [FunctionInvocation](docs/function_invocation.mdx). + - :Function: should only take the first event from the array of events and process it. + - :Function: can be invoked multiple times for multiple events. + +- :AirdropSDKLibrary: for TypeScript helps developers build snap-ins that integrate with DevRev's Airdrop platform. + - The documentation for :AirdropSDKLibrary: is provided in the resource [airdrop_sdk_library_documentation.md](docs/airdrop_sdk_library_documentation.md). - Please note that EventContextIn and EventContextOut interfaces are deprecated. -- The External Domain Metadata is a JSON object that describes the logical domain model of an external system. It defines the types of external record types, the relationships between those record types, and the metadata associated with each field, including the field’s data type, human-readable name, and other descriptive attributes. - - Note: Resource "attachments" *MUST NOT* be included in The External Domain Metadata JSON object even if the external system supports attachments. This is handled automatically by The Airdrop SDK Library. +- :ExternalDomainMetadataJSONObject: is a JSON object that describes the logical domain model of an external system. It defines the types of external record types, the relationships between those record types, and the metadata associated with each field, including the field's data type, human-readable name, and other descriptive attributes. + - Note: Resource "attachments" *MUST NOT* be included in :ExternalDomainMetadataJSONObject: even if the external system supports attachments. This is handled automatically by :AirdropSDKLibrary:. -- The Initial Domain Mapping is a JSON object that defines how the logical domain model described by The External Domain Metadata maps to the target domain model in DevRev. It specifies the correspondence between each external record type and DevRev leaf type, as well as the mapping of individual fields, including transformation logic, value conversions, and any fixed or default values. - - Note: Resource "attachments" *MUST NOT* be included in The Initial Domain Mapping JSON object even if the external system supports attachments. This is handled automatically by The Airdrop SDK Library. +- :InitialDomainMappingJSONObject: is a JSON object that defines how the logical domain model described by :ExternalDomainMetadataJSONObject: maps to the target domain model in DevRev. It specifies the correspondence between each external record type and DevRev leaf type, as well as the mapping of individual fields, including transformation logic, value conversions, and any fixed or default values. + - Note: Resource "attachments" *MUST NOT* be included in :InitialDomainMappingJSONObject: even if the external system supports attachments. This is handled automatically by :AirdropSDKLibrary:. -- The External Sync Unit Item Count is a numeric field representing the total number of items (such as cards) contained within a given external sync unit. It is accessible as "item_count" field of the external sync unit. +- :API: is the API of the external system. -- The Normalization Function is a function that is responsible for normalizing the data from The API to the data expected by The External Domain Metadata JSON object's record type. +- :ExternalSyncUnit: refers to a single unit in the external system that is being synced to DevRev with :SnapIn:. -- The Extraction Function is The Function named "extraction". It is responsible for pushing data to the DevRev servers (The DevRev Servers) based on various event types. +- :APIClient: is a TypeScript service that communicates with :API:. These are the rules for :APIClient: + - If we need to create a new request to :API:, we must create a new method in :APIClient:. + - Communication with :API: must be completely abstracted away from :Function:. :Function: must be able to initialize :APIClient:, call the relevant method from :APIClient: and get the response from :API:. -***Non-Functional Requirements:*** +- :NormalizationFunction: is a function that is responsible for normalizing the data from :API: to the data expected by the record type of :ExternalDomainMetadataJSONObject:. -- The Snap-In must be implemented in TypeScript (The Implementation). +- :DevRevServers: are the servers where :SnapIn: is executed. -- The Snap-In will run on Node.js as the runtime environment. +- :WorkerDataServer: is one of :DevRevServers:. -- The Snap-In should use The Airdrop SDK Library version "1.9.0" for handling data extraction and loading, pushing data, event-driven actions, state management, and artifact handling. +- :TestSnapInServer: is a server used to test :SnapIn: functionality when running :ConformanceTests:. -- Use snake_case for JSON keys. +- :SnapInEventPayload: is a single event that is sent to :SnapIn: server. + - :SnapInEventPayload: is of type `FunctionInput`. + - When sending an :SnapInEventPayload: to :TestSnapInServer:, it is safe to assume that :SnapInEventPayload: will never be undefined, null, or empty. + - When sending an :SnapInEventPayload: to :TestSnapInServer:, you should send a single :SnapInEventPayload: in the request body without wrapping it in an array. -- The Implementation should include unit tests using Jest framework. The required unit test coverage threshold should be set to 60% (but please note that if the unit tests are failing the reported unit test coverage is not correct, that is, it's lower than it actually is). +- :TestCallbackServer: is a server for testing callbacks. -- The code of The Function (including tests) should be in a subfolder at the path `src/functions`. The folder name should correspond to the function name. +- :ExtractionFunction: is :Function: named "extraction". It is responsible reading data from :API: and pushing it to :DevRevServers: based on various event_type values. -- The shared code (including libraries, utilities, adapters, and other common code) used by multiple The Function should be placed in the `src/core` folder. The code in this folder should not depend on any specific The Function implementation. +- :DataExtractionGuide: is provided in the resource [data-extraction.md](docs/data-extraction.md). + - :DataExtractionGuide: should serve as a generic guide for the implementation of :ExtractionFunction:. -- During the data extraction phase, The Normalization Function's "data" field for specific record type should map precisely the fields that are present in The External Domain Metadata JSON object for that record type. The Normalization Function should map no additional fields other than those specified in The External Domain Metadata JSON object, and also no object fields from The External Domain Metadata JSON should be omitted in normalization. +- :DataExtractionRulesForEmittingEvents: is provided in the resource [data-extraction-rules-for-emitting-events.md](docs/data-extraction-rules-for-emitting-events.md). + - :DataExtractionRulesForEmittingEvents: should serve as rules for emitting events for :ExtractionFunction:. +- :AttachedJSONOfTheAcceptanceTest: is a JSON file that is referenced in the resource of :AcceptanceTests:. -***Test Requirements:*** +- :ChefCLI: is command line tool for validating :ExternalDomainMetadataJSONObject:, :InitialDomainMappingJSONObject: and :NormalizationFunction:. + - Path to :ChefCLI: executable is provided in environment variable `CHEF_CLI_PATH`. -- The Conformance Tests of The Snap-In should be implemented in TypeScript with target language version es2017. The Conformance Tests should utilize the TypeScript target language version es2017 with configuration settings for module interoperability, strict type checking, declaration file generation, and JSON module support. +- :ConnectionDataKey: is field `:SnapInEventPayload:["payload"]["connection_data"]["key"]`. -- The Conformance Tests must be in the top folder. +- :ConnectionDataOrgID: is field `:SnapInEventPayload:["payload"]["connection_data"]["org_id"]`. -- The Conformance Tests will run on Node.js as the runtime environment. +- :Credentials: are credentials for :API:. -- The Conformance Tests will be executed using the "npm test" command. The command "npm test" should be accessible in the `package.json` file of The Conformance Tests. +- :ArtifactName: is the name of the artifact being extracted. -- The Conformance Tests must complete within 120 seconds (The Conformance Tests Timeout). If The Conformance Tests Timeout elapses, the process executing The Conformance Tests will be terminated. +- :ExtractionStateObject: is an object representing the state of :ExtractionFunction:. -- The Snap-In runs on Node.js and the test server is accessible at http://localhost:8000/handle/sync (The Test Snap-In Server). +- :InitialExtractionStateObject: is the initial state of :ExtractionStateObject:. + - :InitialExtractionStateObject: should be an instance of :ExtractionStateObject:. + - :InitialExtractionStateObject: should be :ExtractionStateObject: with all "completed" fields set to false, and all the rest of the fields in :ExtractionStateObject: set to undefined, false, 0, or empty string, depending on the type of the field. -- The Conformance Tests should use the The Test Snap-In Server - do not use mock server. +- :CompleteExtractionStateObject: is the complete state of :ExtractionStateObject:. + - :CompleteExtractionStateObject: should be an instance of :ExtractionStateObject:. + - :CompleteExtractionStateObject: should be :ExtractionStateObject: have all the fields set indicating that the extraction is complete for all :ArtifactName:s. -- The Conformance Tests should spin a server at http://localhost:8002 for testing callbacks (The Callback Server). - - The Conformance Tests *MUST NOT* send the event directly to The Callback Server. - - The Implementation Code *MUST NOT* send the event directly to The Callback Server. +- :WorkerThread: is a separate file in which all operations accessing external services or data providers must be implemented. + - :WorkerThread: must be implemented in TypeScript + - :WorkerThread: must be referenced with a .ts (TypeScript) extension, not .js (JavaScript). -- The DevRev Server runs at http://localhost:8003, so it must not be mocked. +- :SpawnMethod: is a method from :AirdropSDKLibrary: that is used to spawn :WorkerThread:. + - :ExtractionFunction: should invoke :SpawnMethod: to spawn :WorkerThread:. + - Here are the rules for invoking :SpawnMethod: + - "isLocalDevelopment" *must not* be set. + - "initialState" should be set to :InitialExtractionStateObject:. -- The Worker Data Server is part of The DevRev Server and is accessible at http://localhost:8003/external-worker, so it must not be mocked. +- :DevRevObject: is a DevRev object. -- The API server is running at http://localhost:8004, so it must not be mocked. +- :DevRevFields: Are fields of the the certain :DevRevObject:. -- The `jest.setup.js` file is managed externally and cannot be modified within The Conformance Tests. +- :ObjectPRD: is a document describing how the :API: objects map to the :DevRevObject: (with corresponding :DevRevFields: mappings). -- All test files from The Conformance Tests must use the `.test.ts` suffix to be discoverable by Jest's default test pattern matching. +***Non-Functional Requirements:*** -- The Conformance Tests should send a single event to the Snap-In Server in the request body without wrapping it in an array. - - It is safe to assume that event payload will never be undefined, null, or empty. +- :Implementation: of :SnapIn: must be in TypeScript. -- Do not invoke The Function with: - - "function_name" that doesn't exist. - - Empty event payload. - - "None" event payload. - - Missing "execution_metadata" field. +- :SnapIn: will run on Node.js as the runtime environment. -- More generally, for any unexpected events sent to The Snap-In Server, do not rely on the returned status codes being correct or meaningful. +- :SnapIn: should use :AirdropSDKLibrary: version "1.9.0" for handling data extraction and loading, pushing data, event-driven actions, state management, and artifact handling. -- If generating The Acceptance Test that references a JSON file (The Attached JSON Of The Acceptance Test) in the resource, you should: - - Store The Attached JSON Of The Acceptance Test in a separate JSON file. - - Make sure to replace placeholders of the credentials (in `event["payload"]["connection_data"]["key"]` and `event["payload"]["connection_data"]["org_id"]`) in The Attached JSON Of The Acceptance Test with the actual values (credentials from The API). +- Use snake_case for JSON keys. -- The Conformance Tests should encourage code reuse. Most notably, you should avoid duplication of the following: - - Setting up The Callback Server, - - Reading environment variables, - - Defining the test event payload sent to The Snap-In Server. +- :Implementation: should include unit tests using Jest framework. The required unit test coverage threshold should be set to 50% (but please note that if the unit tests are failing the reported unit test coverage is not correct, that is, it's lower than it actually is). -# The TypeScript Node.js application boilerplate +- The code of :Function: (including tests) should be in a subfolder at the path `src/functions`. The folder name should correspond to the function name. -***Non-Functional Requirements:*** +- The shared code (including libraries, utilities, adapters, and other common code) used by more than one :Function: should be placed in the `src/core` folder. The code in this folder should not depend on any specific :Function: implementation. -- Logic of pushing data to The DevRev Servers must be implemented in the worker thread and invoked with the "spawn" method from The Airdrop SDK Library. +- During the data extraction phase, the "data" field of :NormalizationFunction: for specific record type should map precisely the fields that are present in :ExternalDomainMetadataJSONObject: for that record type. :NormalizationFunction: should map no additional fields other than those specified in :ExternalDomainMetadataJSONObject:, and also no object fields from :ExternalDomainMetadataJSONObject: should be omitted in normalization. -- The Worker Thread must be implemented in TypeScript and *The Worker Thread MUST BE* referenced with a .ts (TypeScript) extension, not .js (JavaScript). +- When processing an :API: response that contains an array of elements, all subsequent requests to :API: for each element must be executed asynchronously, rather than sequentially (and properly awaited). -- The Function being implemented *MUST not* be called "extraction" or "loading". +- :ExtractionFunction: should not stringify error messages. If an error is thrown, it should be logged before throwing it. -- {% include "templates/should_not_modify_extraction_function.plain" %} +- :ExtractionFunction: is the only :Function: that should invoke :SpawnMethod: to spawn :WorkerThread:. -***Functional Requirements:*** +- :SnapIn: should use `axios` library for making HTTP requests. + +- Requests to :API: *MUST NOT* be mocked. + +***Test Requirements:*** + +- :ConformanceTests: should be implemented in TypeScript with target language version es2017. + - :ConformanceTests: should utilize the TypeScript target language version es2017 with configuration settings for module interoperability, strict type checking, declaration file generation, and JSON module support. + +- :ConformanceTests: will run on Node.js as the runtime environment. + +- :ConformanceTests: will be executed using the "npm test" command. The command "npm test" should be accessible in the `package.json` file of :ConformanceTests:. -- Implement The Function that only checks if The Function can be invoked. +- :ConformanceTests: timeout is 120 seconds. If the timeout elapses, the process executing :ConformanceTests: will be terminated. -- Implement The Function called "test_external_sync_units" (The External Sync Units Test Function) that provides a test of the 'external sync units' part of the extraction workflow as described in the resource [external-sync-units-extraction.mdx](docs/external-sync-units-extraction.mdx). - - If "event_type" equals "EXTRACTION_EXTERNAL_SYNC_UNITS_START", The External Sync Units Test Function should: - - Only emit event "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE". +- :TestSnapInServer: runs on Node.js and is accessible at http://localhost:8000/handle/sync. - ***Acceptance Tests:*** +- :ConformanceTests: should use :TestSnapInServer: - do not use mock server. - - Test The Function using the resource [external_sync_unit_check.json](test_data/external_sync_unit_check.json). Test is successful if The Callback Server receives from DevRev an event with "event_type" that equals "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE". The test must not send the event directly to The Callback Server. +- :ConformanceTests: should spin up :TestCallbackServer: at http://localhost:8002 for testing callbacks. + - :ConformanceTests: *MUST NOT* send :SnapInEventPayload: directly to :TestCallbackServer:. + - :Implementation: *MUST NOT* send :SnapInEventPayload: directly to :TestCallbackServer:. -- Implement The Function called "data_extraction_check" (The Data Extraction Check Function) that provides a test of the 'data extraction' part of the extraction workflow as described in the resource [data-extraction.md](docs/data-extraction.md). The Data Extraction Check Function should: - - if "event_type" equals "EXTRACTION_DATA_START" or "EXTRACTION_DATA_CONTINUE", The Data Extraction Check Function should: - - Initialize a single repo "users" - - normalize the users data using The Normalization Function - - push the normalized users data to The DevRev Servers - - Emit event "EXTRACTION_DATA_DONE". - - Note: The Data Extraction Check Function should be simplified and should disregard all other details. +- :DevRevServers: run at http://localhost:8003 and they must not be mocked. + +- :WorkerDataServer: is accessible at http://localhost:8003/external-worker and it must not be mocked. + +- File `jest.setup.js` is managed externally and cannot be modified within :ConformanceTests:. + +- All test files from :ConformanceTests: must use the `.test.ts` suffix to be discoverable by Jest's default test pattern matching. + +- :ConformanceTests: should send a single :SnapInEventPayload: to :TestSnapInServer: in the request body without wrapping it in an array. + - It is safe to assume that :SnapInEventPayload: will never be undefined, null, or empty. + +- If using :AttachedJSONOfTheAcceptanceTest: in :AcceptanceTests:, you should: + - Store :AttachedJSONOfTheAcceptanceTest: in a separate JSON file. + - Make sure to replace placeholders of the credentials (in `event["payload"]["connection_data"]["key"]` and `event["payload"]["connection_data"]["org_id"]`) in :AttachedJSONOfTheAcceptanceTest: with the actual values (credentials from :API:). + +- :ConformanceTests: should encourage code reuse. Most notably, you should avoid duplication of the following: + - Setting up :TestCallbackServer:, + - Reading environment variables, + - Defining the test :SnapInEventPayload: sent to :TestSnapInServer:. + +- :ConformanceTests: *MUST NOT*: + - Call :DevRevServers: directly, unless explicitly instructed to do so. + - Retrieve artifacts from :DevRevServers: directly and write tests based on them. + +- :ConformanceTests: *MUST NOT* generate the following tests (unless explicitly instructed to do so): + - Tests that have incomplete :SnapInEventPayload: payloads + - You can assume that :SnapInEventPayload: sent to :TestSnapInServer: is valid and complete. + - You need not test situations with incomplete :SnapInEventPayload: payloads (empty, None, missing fields, etc.). + - Tests with rate limiting (rate limiting will be handled through :AcceptanceTests:). + - If test is testing :ExtractionFunction:, it should not do retries on the snap-in server and invoke :Function: multiple times. + - Tests that rely on status codes for information of success. + - Tests that validate :NormalizationFunction: with :ChefCLI: (unless explicitly instructed to do so). + +- If :ConformanceTests: is testing "data extraction" part of :ExtractionFunction:, it should create *at most* one test for validating the data extraction logic. + - If previous :ConformanceTests: already exists for invoking "data extraction" part of :ExtractionFunction:, you need not create a test for it once again. + +- If :ConformanceTests: is testing "metadata extraction" part of :ExtractionFunction:, :ConformanceTests: *MUST NOT* test the contents of :ExternalDomainMetadataJSONObject: against the expected structure and contents. + +# The TypeScript Node.js application boilerplate + +***Functional Requirements:*** - ***Acceptance Tests:*** - - - Test The Function using the resource [data_extraction_check.json](test_data/data_extraction_check.json). Test is successful if The Callback Server receives from DevRev an event with "event_type" that equals "EXTRACTION_DATA_DONE". The test must not send the event directly to The Callback Server. +- Implement :Function: "health_check" that only checks if :Function: can be invoked. \ No newline at end of file diff --git a/templates/attachment_extraction_acceptance_test.plain b/templates/attachment_extraction_acceptance_test.plain index 156a185..0aeeb46 100644 --- a/templates/attachment_extraction_acceptance_test.plain +++ b/templates/attachment_extraction_acceptance_test.plain @@ -1,10 +1,10 @@ -Test The Extraction Function with the following flow (The Attachment Test Flow): - - Step 1: Invoke The Extraction Function using the resource [{{ data_extraction_resource_name }}](test_data/{{ data_extraction_resource_name }}). - - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". - - Step 2: Invoke The Extraction Function using the resource [{{ attachment_extraction_resource_name }}](test_data/{{ attachment_extraction_resource_name }}). - - Expect The Callback Server to receive from DevRev a **single** event (`:= event_from_callback_server`) with "event_type" that equals "EXTRACTION_ATTACHMENTS_DONE". +Test :ExtractionFunction: with the following flow (`:= attachmentTestFlow`): + - Step 1: Invoke :ExtractionFunction: using the resource [{{ data_extraction_resource_name }}](test_data/{{ data_extraction_resource_name }}). + - Expect :TestCallbackServer: to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". + - Step 2: Invoke :ExtractionFunction: using the resource [{{ attachment_extraction_resource_name }}](test_data/{{ attachment_extraction_resource_name }}). + - Expect :TestCallbackServer: to receive from DevRev a **single** event (`:= event_from_callback_server`) with "event_type" that equals "EXTRACTION_ATTACHMENTS_DONE". - Expect `event_from_callback_server["event_data"]["artifacts"]` to be an array. Expect this array to not be empty. Expect this array to have length 1. - Let `artifact_object = event_from_callback_server["event_data"]["artifacts"][0]`. - Expect `artifact_object["item_type"] = "ssor_attachment"` and `artifact_object["item_count"] = {{ expected_attachment_count }}`. - Make a GET request to "http://localhost:8003/is_uploaded/{artifact_object['id']}". Expect response code to be equal to 200. - - Note: The Attachment Test Flow should be executed within a single test. \ No newline at end of file + - Note: attachmentTestFlow should be executed within a single test. \ No newline at end of file diff --git a/templates/chef_cli_normalization_validation.plain b/templates/chef_cli_normalization_validation.plain index 101f428..6a86655 100644 --- a/templates/chef_cli_normalization_validation.plain +++ b/templates/chef_cli_normalization_validation.plain @@ -1,10 +1,14 @@ -validate The Normalization Function with the chef-cli tool (The Chef CLI) using the command "chef-cli validate-data -m -r {{ record_type }}": - - To invoke the function that returns The External Domain Metadata JSON object, send the event [external_domain_metadata_event_payload.json](test_data/external_domain_metadata_event_payload.json) to The Snap-In server. - - The External Domain Metadata JSON object should then be stored in temporary file and provided as path to the validation of The Normalization Function. - - The Chef CLI reads the extracted file (The Extracted File) from stdin. - - The Extracted File is inside folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH". To retrieve the relevant file name, command `ls EXTRACTED_FILES_FOLDER_PATH | grep extractor_{{ record_type }} | sort -r | head -n 1` should be executed. The test should always fail if The Extracted File is not present. - - Please note that the expected behaviour is that only after the invocation of The Extraction Function, the folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH" should exist. The test should always fail if before the execution of The Chef CLI, the folder "EXTRACTED_FILES_FOLDER_PATH" does not exist or if The Extracted File is not found. - - The path to The Chef CLI executable is provided in the environment variable `CHEF_CLI_PATH`. - - For the validation of The Normalization Function to be successful The Chef CLI must return an empty output. The implementation of The Extraction Function directly infuences the output of The Chef CLI. - - The stdout and stderr of The Chef CLI *must* be printed to the console. - - The test should always fail if The Chef CLI is not available. \ No newline at end of file +Validate :NormalizationFunction: with :ChefCLI: for the "{{ record_type }}" record type with the following flow: + - Step 1.: Invoke :ExtractionFunction: using the resource [data_extraction_test.json](test_data/data_extraction_test.json). + - Step 2.: Retrieve full extracted file path (`:= fullExtractedFilePath`) + - The extracted file name (`:= extractedFileName`) is the name of the file created by :ExtractionFunction: + - extractedFileName is located in the folder provided in the environment variable "EXTRACTED_FILES_FOLDER_PATH". + - To retrieve extractedFileName, execute the following command: `ls EXTRACTED_FILES_FOLDER_PATH | grep extractor_{{ record_type }} | sort -r | head -n 1` + - The test should fail the folder "EXTRACTED_FILES_FOLDER_PATH" does not exist or if the extractedFileName is not found. + - fullExtractedFilePath is the full path to the extractedFileName. + - Step 3.: In :ConformanceTests:, create a separate file for :ExternalDomainMetadataJSONObject:. Contents of :ExternalDomainMetadataJSONObject: JSON object **MUST NOT** be modified compared to the original file contents. + - Step 4.: Validate :NormalizationFunction: with :ChefCLI: using the command `"${CHEF_CLI_PATH}" validate-data -m -r {{ record_type }} < `: + - Validation is successful precisely when both the stdout and stderr of :ChefCLI: are empty. + - Expect the validation of :ChefCLI: to succeed. If it fails, normalization method for "{{ record_type }}" in :Implementation: should be fixed. + - For the validation of :NormalizationFunction: to be successful The :ChefCLI: must return an empty output. Note that the implementation of :ExtractionFunction: directly infuences the output of :ChefCLI:. + - The stdout and stderr of :ChefCLI: *MUST* be printed to the console. \ No newline at end of file diff --git a/templates/data_extraction_acceptance_test.plain b/templates/data_extraction_acceptance_test_no_pagination.plain similarity index 69% rename from templates/data_extraction_acceptance_test.plain rename to templates/data_extraction_acceptance_test_no_pagination.plain index c517aca..2f2fa85 100644 --- a/templates/data_extraction_acceptance_test.plain +++ b/templates/data_extraction_acceptance_test_no_pagination.plain @@ -1,5 +1,5 @@ -Test The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). - - Expect The Callback Server to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". +Test :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}). + - Expect :TestCallbackServer: to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". - Let `artifact_array = ["event_data"]["artifacts"]` (array of objects). - Expect `len(artifact_array) > 0`. - Out of `artifact_array`, expect one of the elements to have "item_type" equal to "{{ artifact_name }}" (`:= {{ artifact_name }}_artifact`) and `{{ artifact_name }}_artifact["item_count"]={{ expected_item_count }}`. diff --git a/templates/data_extraction_acceptance_test_with_pagination.plain b/templates/data_extraction_acceptance_test_with_pagination.plain new file mode 100644 index 0000000..e8e197c --- /dev/null +++ b/templates/data_extraction_acceptance_test_with_pagination.plain @@ -0,0 +1,7 @@ +Test :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}). + - Expect :TestCallbackServer: to receive from DevRev a **single** event with "event_type" that equals "EXTRACTION_DATA_DONE". + - Let `artifact_array = ["event_data"]["artifacts"]` (array of objects). + - Expect `len(artifact_array) > 0`. + - Out of `artifact_array`, expect one of the elements to have "item_type" equal to "{{ artifact_name }}" (`:= {{ artifact_name }}_artifact`) and `{{ artifact_name }}_artifact["item_count"]={{ expected_item_count }}`. + - If `{{ artifact_name }}_artifact["item_count"] < {{ expected_item_count }}`, this indicates that not all the {{ artifact_name }} data was extracted. + - If `{{ artifact_name }}_artifact["item_count"] < {{ expected_item_count }}` and equals {{ fetched_page_size }}, it means :ExtractionFunction: fetched only one page instead of all pages. Pages should be fetched iteratively until all pages are fetched. \ No newline at end of file diff --git a/templates/data_fetching_function_resource.plain b/templates/data_fetching_function_resource.plain new file mode 100644 index 0000000..4161a50 --- /dev/null +++ b/templates/data_fetching_function_resource.plain @@ -0,0 +1 @@ +Implement the :Function: "fetch_{{ record_type }}" that fetches all {{ record_type }} as specified in :ObjectPRD: [{{ record_type }}-extraction.md](product_requirements/{{ record_type }}-extraction.md) using {{ pagination_rule }}. \ No newline at end of file diff --git a/templates/data_fetching_return_expectation.plain b/templates/data_fetching_return_expectation.plain index 1e92e62..513c50f 100644 --- a/templates/data_fetching_return_expectation.plain +++ b/templates/data_fetching_return_expectation.plain @@ -1,5 +1,5 @@ -The Functions implemented *should* also return: - - `status_code` - a status code from The API. - - `api_delay` - Delay in seconds to wait before retrying the request to The API (if no rate limit, set to 0) - - `message` - Information about the call the The Function. - - Note: Every The Function should handle rate limiting. \ No newline at end of file +:Function: implemented *should* also return: + - `status_code` - a status code from :API:. + - `api_delay` - Delay in seconds to wait before retrying the request to :API: (if no rate limit, set to 0) + - `message` - Information about the call :Function:. + - `data` (optional) - Data returned from :API:. \ No newline at end of file diff --git a/templates/edm_record_type_template.plain b/templates/edm_record_type_template.plain new file mode 100644 index 0000000..e6598b0 --- /dev/null +++ b/templates/edm_record_type_template.plain @@ -0,0 +1 @@ +The :Function: 'get_external_domain_metadata' should return :ExternalDomainMetadataJSONObject: with the record type "{{ record_type }}" based on external fields from file [{{ product_requirement_file }}]({{ product_requirement_file }}), while preserving any existing record types. \ No newline at end of file diff --git a/templates/external_domain_metadata_boilerplate.plain b/templates/external_domain_metadata_boilerplate.plain index 07b7cf7..36c32f4 100644 --- a/templates/external_domain_metadata_boilerplate.plain +++ b/templates/external_domain_metadata_boilerplate.plain @@ -1,30 +1,36 @@ ***Definitions:*** -- The structure of The External Domain Metadata JSON object is specified by the JSON schema defined in the resource [external_domain_metadata_schema.json](external_domain_metadata_schema.json). +- The structure of :ExternalDomainMetadataJSONObject: is specified by the JSON schema defined in the resource [external_domain_metadata_schema.json](external_domain_metadata_schema.json). - Please note that all refers_to keys in reference and typed_reference fields must use the format "#record:", where matches a key in the top-level record_types. This ensures references are unambiguous and valid. Use {} as the value to refer by default identifier (id), or specify { "by_field": "field_name" } to use a custom key. - A “display name” is the human-readable label of a field within a given record type. ***Non-Functional Requirements:*** -- Store The External Domain Metadata JSON object as a separate JSON file. +- Store :ExternalDomainMetadataJSONObject: as a separate JSON file. -- The External Domain Metadata JSON object *MUST* be imported without "as" statement in The Implementation Code. +- :ExternalDomainMetadataJSONObject: *MUST* be imported without "as" statement in :Implementation:. -- {% include "templates/should_not_modify_extraction_function.plain" %} +- The :UnitTests: *MUST NOT* test the contents and *MUST NOT* teset the of :ExternalDomainMetadataJSONObject:. ***Test Requirements:*** -- Validate generated The External Domain Metadata JSON object with the chef-cli tool (The Chef CLI) using the command "chef-cli validate-metadata": +- Validate generated :ExternalDomainMetadataJSONObject: with :ChefCLI: using the command "${CHEF_CLI_PATH} validate-metadata": - This command reads JSON object from stdin. - - The path to The Chef CLI executable is provided in the environment variables `CHEF_CLI_PATH`. - - For the validation of The External Domain Metadata JSON object to be successful The Chef CLI must return an empty output. - - The test should always fail if The Chef CLI is not available. + - The path to :ChefCLI: executable is provided in the environment variables `CHEF_CLI_PATH`. + - For the validation of :ExternalDomainMetadataJSONObject: to be successful :ChefCLI: must return an empty output. + - The test should always fail if :ChefCLI: is not available. -- The External Domain Metadata JSON object should always be read from The Implementation Code. +- :ExternalDomainMetadataJSONObject: *MUST* be validated solely against :ChefCLI:. + - You *MUST NOT* validate :ExternalDomainMetadataJSONObject: against the literal JSON contents of :ExternalDomainMetadataJSONObject:. + - :ConformanceTests: should have *ONLY TWO* tests + - That :Function: returning :ExternalDomainMetadataJSONObject: can be invoked successfully + - :ChefCLI: validation using the command "${CHEF_CLI_PATH} validate-metadata" -- If The Conformance Tests test The External Domain Metadata JSON object, test it only against The Chef CLI. Don't test it against the literal JSON contents of The External Domain Metadata JSON object. +- :ExternalDomainMetadataJSONObject: should always be read from :Implementation: (and not recreated in :ConformanceTests:) -- The stdout and stderr of The Chef CLI *must* be printed to the console. +- The stdout and stderr of :ChefCLI: *must* be printed to the console. -- The Conformance Tests should not modify nor do any transformations on The External Domain Metadata JSON object at all. If The Conformance Tests indicates a problem with The External Domain Metadata JSON object, The External Domain Metadata JSON object in The Implementation Code should be adjusted to fix the problem. The tests should only validate the JSON object in its original form. \ No newline at end of file +- :ConformanceTests: should not modify nor do any transformations on :ExternalDomainMetadataJSONObject: at all. + - If :ConformanceTests: indicates a problem with :ExternalDomainMetadataJSONObject:, :ExternalDomainMetadataJSONObject: in :Implementation: should be adjusted to fix the problem. + - :ConformanceTests: should only validate the JSON object in its original form. \ No newline at end of file diff --git a/templates/external_sync_unit_acceptance_test.plain b/templates/external_sync_unit_acceptance_test.plain index 0a24cda..a20dbab 100644 --- a/templates/external_sync_unit_acceptance_test.plain +++ b/templates/external_sync_unit_acceptance_test.plain @@ -1,5 +1,5 @@ -Test The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). - - Expect The Callback Server to receive *a single* event with "event_type" "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE". +Test :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}). + - Expect :TestCallbackServer: to receive *a single* event with "event_type" "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE". - Let `external_sync_units_array = ["event_data"]["external_sync_units"]` (array of objects). - Expect `external_sync_units_array` to exist and be an array. - Expect `len(external_sync_units_array) = {{ expected_external_sync_unit_count }}`. diff --git a/templates/extraction_function_resource.plain b/templates/extraction_function_resource.plain new file mode 100644 index 0000000..816d737 --- /dev/null +++ b/templates/extraction_function_resource.plain @@ -0,0 +1,6 @@ +If "event_type" equals "EXTRACTION_DATA_START" or "EXTRACTION_DATA_CONTINUE" :ExtractionFunction: should extract the '{{ record_type }}' data, while preserving previous artifact pushes of data. + - Refer to resource [{{ record_type }}-extraction.md](product_requirements/{{ record_type }}-extraction.md) for details on {{ record_type }} extraction specification. + - Refer to resource [high-level-data-extraction-plan.md](product_requirements/high-level-data-extraction-plan.md) for: + - The order of artifact extraction. + - Information if we should use pagination or not for the extraction of the '{{ record_type }}' data. + - {{additional_instructions}} \ No newline at end of file diff --git a/templates/initial_domain_mapping_boilerplate.plain b/templates/initial_domain_mapping_boilerplate.plain deleted file mode 100644 index 2f9d303..0000000 --- a/templates/initial_domain_mapping_boilerplate.plain +++ /dev/null @@ -1,50 +0,0 @@ -***Definitions:*** - -- The structure of The Initial Domain Mapping JSON object is specified by the JSON schema defined in the resource [initial_mappings_schema.yaml](initial_mappings_schema.yaml). - - For a complete list of supported DevRev object types and their fields, see resource [Supported DevRev object types for Airdrop](docs/supported-object-types.md). - - For information about transformation methods, see resource [Mapping Reasons](docs/mapping-reasons.mdx). - - When working with devrev_leaf_type, be aware that the schema expects different formats depending on context. In most places, it should be passed as an object with object_category and object_type. However, in the airdrop-recipe-create-possible-record-type-mapping context, it must be a string representing only the object_type. - - Please note that mappings are split into separate ‘shards’ - one for each record type - for easier manipulation and storage. - - Please note that a leaf type or a concrete id has to be selected for use_devrev_record, but not both. - - Please note that field "record_type_mappings" has to be placed inside the "additional_mappings" object. - -- The Stock Field Mapping Field is a configuration object within the Initial Domain Mapping JSON structure that defines how external system data fields are mapped to DevRev's built-in (stock) fields. It is located within the "stock_field_mappings" object inside a blueprint shard configuration. - -- The Fixed Transformation Method is a transformation method that is used to set a fixed value to a field. To apply The Fixed Transformation Method to The Stock Field Mapping Field, set the Metadata extraction transformation method type to "use_fixed_value". If using The Fixed Transformation Method, the value of the field "transformation_method_for_set" inside The Stock Field Mapping Field should be `{"enum": "", "transformation_method": "use_fixed_value", "value": "enum_value"}`, where `` is the fixed value. - -- The External Transformation Method is a transformation method that is used to map an external field to a DevRev stock field. To apply The External Transformation Method to The Stock Field Mapping Field, set the Metadata extraction transformation method type to "use_directly", "use_as_array_value", "use_rich_text", depending on the context. Field "primary_external_field" should be set to the name of the external field to map. - -- The Map Enum Transformation Method is a transformation method that is used to map an external field enum to a DevRev enum. To apply The Map Enum to The Stock Field Mapping Field, set the Metadata extraction transformation method type to "map_enum". If using The Map Enum, the value of the field "transformation_method_for_set" inside The Stock Field Mapping Field should be `{"forward": "", "transformation_method": "map_enum", "is_array": false}`, where `` is a JSON object. This JSON object has keys, which are all possible enum values of the external field. Each key has a value, which is also a JSON object `{"value": ""}`, where `` is the enum value to which the external field enum maps. - -- The DevRev Record Transformation Method is a transformation method that is used enable use of a fixed reference to something in DevRev. To apply The DevRev Record Transformation Method to The Stock Field Mapping Field, set the Metadata extraction transformation method type to "use_devrev_record". - -- The Shard Mode Type is an option in The Initial Domain Mapping JSON object through which you can specify shard type for the mapping for the given record type. - - You can either create a fresh shard to use for this mapping, assign an existing one. - -- For every record type in The Initial Domain Mapping JSON object, The Shard Mode Type should be set to "create_shard". - -***Non-Functional Requirements:*** - -- Store The Initial Domain Mapping JSON object as a separate JSON file. - -- The External Domain Metadata JSON object should not be modified. If there are discrepancies between The External Domain Metadata JSON object and The Initial Domain Mapping JSON object, assume The External Domain Metadata JSON object is correct and The Initial Domain Mapping JSON object needs to be adjusted. - -- {% include "templates/should_not_modify_extraction_function.plain" %} - -***Test Requirements:*** - -- Validate generated The Initial Domain Mapping JSON object with the chef-cli tool (The Chef CLI) using the command "chef-cli initial-mapping check -m ": - - The Chef CLI reads The Initial Domain Mapping JSON object from stdin. - - For return type of The Chef CLI to stdout, see resource [chef_cli_output_schema.json](resources/chef_cli_output_schema.json). Fields "RemainingDeficiencies" and "Warnings" contains information about the deficiencies that need to be resolved for the validation to pass. - - The path to The Chef CLI executable is provided in the environment variables `CHEF_CLI_PATH`. - - The test should always fail if we get an empty output. - - The test must fail unless both of the fields "RemainingDeficiencies" and "Warnings" of the first object in the Chef CLI output array are present and both of their values are exactly null. - - The test should always fail if The Chef CLI is not available. - -- The Initial Domain Mapping JSON object should always be read from The Implementation Code. - -- The stdout and stderr of The Chef CLI *must* be printed to the console. - -- If The Conformance Tests test The Initial Domain Mapping JSON object, test it only against The Chef CLI. Don't test it against the literal JSON contents of The Initial Domain Mapping JSON object. - -- The Conformance Tests should not modify nor do any transformations on The Initial Domain Mapping JSON object at all. If The Conformance Tests indicates a problem with The Initial Domain Mapping JSON object, The Initial Domain Mapping JSON object in The Implementation Code should be adjusted to fix the problem. The tests should only validate the JSON object in its original form. diff --git a/templates/internal_client.plain b/templates/internal_client.plain deleted file mode 100644 index a41807e..0000000 --- a/templates/internal_client.plain +++ /dev/null @@ -1,3 +0,0 @@ -The {{ external_system_name }} Internal Client is a TypeScript service that communicates with The API. These are the rules for The {{ external_system_name }} Internal Client: - - If we need to create a new request to The API, we must create a new method in The {{ external_system_name }} Internal Client. - - Communication with The API must be completely abstracted away from The Function. The Function must be able to initialize The {{ external_system_name }} Internal Client, call the relevant method from The {{ external_system_name }} Internal Client and get the response from The API. \ No newline at end of file diff --git a/templates/should_not_modify_extraction_function.plain b/templates/should_not_modify_extraction_function.plain index f296b2b..1884175 100644 --- a/templates/should_not_modify_extraction_function.plain +++ b/templates/should_not_modify_extraction_function.plain @@ -1 +1 @@ -The Extraction Function *must not* be modified. \ No newline at end of file +:ExtractionFunction: *must not* be modified. \ No newline at end of file diff --git a/templates/spawn_method_instructions.plain b/templates/spawn_method_instructions.plain deleted file mode 100644 index e421ff5..0000000 --- a/templates/spawn_method_instructions.plain +++ /dev/null @@ -1,11 +0,0 @@ -- All operations accessing external services or data providers must be implemented in worker thread (The Worker Thread). The Worker Thread should be initialized with the method "spawn" (The Spawn Method). The Spawn Method should be initialized with the following parameters: - - The Extraction Function should start The Worker Thread with timeout of 10 minutes. - - "isLocalDevelopment" *must not* be set. - - "initialState" should be The Extraction State Object with all "completed" fields set to false, and all the rest of the fields in The Extraction State Object set to undefined, false, 0, or empty string, depending on the type of the field. - - Note: No additional fields should be added to The Extraction State Object. - - Note: The Worker Thread must be implemented in TypeScript and *The Worker Thread MUST BE* referenced with a .ts (TypeScript) extension, not .js (JavaScript). - -- Use The Initial Domain Mapping JSON object for initialDomainMapping parameter when spawning a new worker. - - Note: The Initial Domain Mapping JSON object should be read directly from the JSON file. - -- The Extraction Function should not stringify error messages. If an error is thrown, it should be logged before throwing it. \ No newline at end of file diff --git a/templates/test_rate_limiting_during_data_extraction.plain b/templates/test_rate_limiting_during_data_extraction.plain index 64ed389..b6fa42a 100644 --- a/templates/test_rate_limiting_during_data_extraction.plain +++ b/templates/test_rate_limiting_during_data_extraction.plain @@ -1,7 +1,6 @@ -Test the rate limiting of The Function "{{ function_name }}" with the following flow: +Test the rate limiting of :Function: "{{ function_name }}" with the following flow: - Make a POST request to "http://localhost:8004/start_rate_limiting" with body `{ "test_name": }`. - - Invoke The Function "{{ function_name }}" with valid credentials and all required parameters. + - Invoke :Function: "{{ function_name }}" with valid credentials and all required parameters. - Expect: `status_code = 429`. - - Expect: `api_delay > 0` and `api_delay <= 3`. If api_delay is bigger than 3, assume the problem is that we did not calculate the api_delay correctly in The Implementation Code. - - Make a POST request to "http://localhost:8004/end_rate_limiting". + - Expect: `api_delay > 0` and `api_delay <= 3`. If api_delay is bigger than 3, assume the problem is that we did not calculate the api_delay correctly in :Implementation:. \ No newline at end of file diff --git a/templates/test_rate_limiting_during_emitting_events.plain b/templates/test_rate_limiting_during_emitting_events.plain index 7744647..d3d2be1 100644 --- a/templates/test_rate_limiting_during_emitting_events.plain +++ b/templates/test_rate_limiting_during_emitting_events.plain @@ -1,5 +1,4 @@ -When the input event is "{{ input_event_name }}", the Extraction Function must handle rate limiting using the following test flow: +When the input event is "{{ input_event_name }}", :ExtractionFunction: must handle rate limiting using the following test flow: - Step 1: Make a POST request to "http://localhost:8004/start_rate_limiting" with body `{ "test_name": }`. - - Step 2: Invoke The Extraction Function using the resource [{{ resource_name }}](test_data/{{ resource_name }}). - - Expect The Callback Server to receive *a single* event with "event_type" "{{ expected_output_event_type }}". - - Step 3: Make a POST request to "http://localhost:8004/end_rate_limiting". \ No newline at end of file + - Step 2: Invoke :ExtractionFunction: using the resource [{{ resource_name }}](test_data/{{ resource_name }}). + - Expect :TestCallbackServer: to receive *a single* event with "event_type" "{{ expected_output_event_type }}". \ No newline at end of file