Updated shared files (#16)

This PR includes a couple of changes regarding the shared files for code
generation with Codeplain (updates to docs, templates, updates to
template Plain file, etc.).

Related issue: https://app.devrev.ai/devrev/works/ISS-201997

---------

Co-authored-by: Dusan Omercevic <dusan.omercevic@gmail.com>
Co-authored-by: Patricija Brečko <patricija.brecko@devrev.ai>

Author: 3 people

base_folder/src/core/types.ts

@@ -0,0 +1,52 @@
+/**
+ * Type definitions for DevRev function inputs and related types
+ */
+
+export type Context = {
+    // ID of the dev org for which the function is being invoked.
+    dev_oid: string;
+    // ID of the automation/command/snap-kit Action/Event Source for which the function is being invoked.
+    source_id: string;
+    // ID of the snap-in as part of which the function is being invoked.
+    snap_in_id: string;
+    // ID of the snap-in Version as part of which the function is being invoked.
+    snap_in_version_id: string;
+    // ID of the service account.
+    service_account_id: string;
+    // This secrets map would contain some secrets which platform would provide to the snap-in.
+    // `service_account_token`: This is the token of the service account which belongs to this snap-in. This can be used to make API calls to DevRev. 
+    // `actor_session_token`: For commands, and snap-kits, where the user is performing some action, this is the token of the user who is performing the action.
+    secrets: Record<string, string>;
+  };
+  
+  export type ExecutionMetadata = {
+    // A unique id for the function invocation. Can be used to filter logs for a particular invocation.
+    request_id: string;
+    // Function name as defined in the manifest being invoked.
+    function_name: string;
+    // Type of event that triggered the function invocation as defined in manifest.
+    event_type: string;
+    // DevRev endpoint to which the function can make API calls.
+    // Example : "https://api.devrev.ai/"
+    devrev_endpoint: string;
+  };
+  
+  export type InputData = {
+    // Map of organization inputs and their corresponding values stored in snap-in.
+    // The values are passed as string and typing need to be handled by the function
+    global_values: Record<string, string>;
+    // Map of event sources and their corresponding ids stored in snap-in.
+    // These could be used to schedule events on a schedule based event source.
+    event_sources: Record<string, string>;
+  };
+  
+  // Event sent to our app.
+  export type FunctionInput = {
+    // Actual payload of the event.
+    payload: Record<string, any>;
+    // Context of the function invocation.
+    context: Context;
+    // Metadata of the function invocation.
+    execution_metadata: ExecutionMetadata;
+    input_data: InputData;
+  };

base_folder/src/core/utils.ts

@@ -0,0 +1,30 @@
+import { AirdropEvent, AirdropMessage } from '@devrev/ts-adaas';
+import { FunctionInput } from './types';
+
+export function convertToAirdropEvent(fi: FunctionInput): AirdropEvent {
+    // Create a properly structured AirdropMessage
+    const airdropMessage: AirdropMessage = {
+      connection_data: fi.payload.connection_data,
+      event_context: {
+        ...fi.payload.event_context,
+        ...fi.context,
+        request_id: fi.execution_metadata?.request_id,
+      },
+      event_type: fi.payload.event_type,
+      event_data: fi.payload.event_data || {},
+    };
+    
+    return {
+      context: {
+        ...fi.context,
+        secrets: {
+          service_account_token: fi.context.secrets?.service_account_token || '',
+          ...fi.context.secrets,
+        },
+      },
+      payload: airdropMessage,
+      execution_metadata: fi.execution_metadata,
+      input_data: fi.input_data,
+    };
+  }
+  

devrev-snapin-template.plain

@@ -4,6 +4,7 @@
 
 - The Function implements part of The Snap-In functionality.
   - Details on how The Function must be invoked are provided in the resource [Function Invocation](docs/function_invocation.mdx).
+  - The Function should only take the first event from the array of events and process it. The Function can be invoked multiple times for multiple events.
 
 - The Airdrop SDK Library for TypeScript helps developers build snap-ins that integrate with DevRev’s Airdrop platform.
   - The documentation for the The Airdrop SDK Library is provided in the resource [airdrop_sdk_library_documentation.md](docs/airdrop_sdk_library_documentation.md).
@@ -13,6 +14,12 @@
 
 - The Initial Domain Mapping is a JSON object that defines how the logical domain model described by The External Domain Metadata maps to the target domain model in DevRev. It specifies the correspondence between each external record type and DevRev leaf type, as well as the mapping of individual fields, including transformation logic, value conversions, and any fixed or default values.
 
+- The External Sync Unit Item Count is a numeric field representing the total number of items (such as cards) contained within a given external sync unit. It is accessible as "item_count" field of the external sync unit.
+
+- The Normalization Function is a function that is responsible for normalizing the data from The API to the data expected by The External Domain Metadata JSON object's record type.
+
+- The Extraction Function is The Function named "extraction". It is responsible for pushing data to the DevRev servers (The DevRev Servers) based on various event types.
+  - "event_type" equals "EXTRACTION_DATA_START" or "event_type" equals "EXTRACTION_DATA_CONTINUE" and we get a timeout, The Worker Thread should emit event "EXTRACTION_DATA_PROGRESS".
 
 ***Non-Functional Requirements:***
 
@@ -21,8 +28,6 @@
 - The Snap-In will run on Node.js as the runtime environment.
 
 - The Snap-In should use The Airdrop SDK Library version "1.5.1" for handling data extraction and loading, pushing data, event-driven actions, state management, and artifact handling.
-  - Please note that 'processTask' function **only** executes task if 'processTask' function is called from a worker thread. Therefore 'processTask' function should be called from a worker thread spawned using 'Spawn' class.
-  - If the worker is written in TypeScript, it must be referenced with a .ts extension, not .js.
 
 - Use snake_case for JSON keys.
 
@@ -32,16 +37,18 @@
 
 - The shared code (including libraries, utilities, adapters, and other common code) used by multiple The Function should be placed in the `src/core` folder. The code in this folder should not depend on any specific The Function implementation.
 
+- During the data extraction phase, The Normalization Function's "data" field for specific record type should map precisely the fields that are present in The External Domain Metadata JSON object for that record type. The Normalization Function should map no additional fields other than those specified in The External Domain Metadata JSON object, and also no object fields from The External Domain Metadata JSON should be omitted in normalization.
+
 
 ***Test Requirements:***
 
-- The Conformance Tests of The Snap-In should be implemented in TypeScript.
+- The Conformance Tests of The Snap-In should be implemented in TypeScript with target language version es2017. The Conformance Tests should utilize the TypeScript target language version es2017 with configuration settings for module interoperability, strict type checking, declaration file generation, and JSON module support.
 
 - The Conformance Tests must be in the top folder.
 
 - The Conformance Tests will run on Node.js as the runtime environment.
 
-- The Conformance Tests will be executed using the "npm test" command.
+- The Conformance Tests will be executed using the "npm test" command. The command "npm test" should be accessible in the `package.json` file of The Conformance Tests.
 
 - The Conformance Tests must complete within 120 seconds (The Conformance Tests Timeout). If The Conformance Tests Timeout elapses, the process executing The Conformance Tests will be terminated.
 
@@ -55,28 +62,36 @@
 
 - The Worker Data Server is part of The DevRev Server and is accessible at http://localhost:8003/external-worker, so it must not be mocked.
 
+- The `jest.setup.js` file is managed externally and cannot be modified within The Conformance Tests.
 
-# The TypeScript Node.js application boilerplate
+- All test files from The Conformance Tests must use the `.test.ts` suffix to be discoverable by Jest's default test pattern matching.
 
-***Functional Requirements:***
+- The Conformance Tests should send a single event to the Snap-In Server in the request body without wrapping it in an array.
 
-- Implement The Function that only checks if The Function can be invoked.
 
-- Implement The Function that only checks if the data extraction workflow can be invoked.
+# The TypeScript Node.js application boilerplate
 
-  ***Acceptance Tests:***
+***Non-Functional Requirements:***
+
+- Logic of pushing data to The DevRev Servers must be implemented in the worker thread and invoked with the "spawn" method from The Airdrop SDK Library.
 
-  - Test the data extraction workflow invocation using the resource [extraction_health_check.json](test_data/extraction_health_check.json).
+- The Function being implemented *MUST not* be called "extraction" or "loading".
 
-- Implement The Function that checks if pushing data works. The URL where data is to be pushed is provided in the "callback_url" field of the "event_context" dictionary which is part of "payload" dictionary.
+***Functional Requirements:***
+
+- Implement The Function that only checks if The Function can be invoked.
 
-- Implement The Function that provides a test of the 'external sync units' part of the extraction workflow. The name of The Function must be 'extraction_external_sync_unit_check'.
+- Implement The Function called "test_external_sync_units" that provides a test of the 'external sync units' part of the extraction workflow as described in the resource [external-sync-units-extraction.mdx](docs/external-sync-units-extraction.mdx). Only check if the extraction can be triggered and that it successfully completes. Disregard all other details.
 
   ***Acceptance Tests:***
 
   - Test The Function using the resource [external_sync_unit_check.json](test_data/external_sync_unit_check.json). Test is successful if The Callback Server receives from DevRev an event with "event_type" that equals "EXTRACTION_EXTERNAL_SYNC_UNITS_DONE". The test must not send the event directly to The Callback Server.
 
-- Implement The Function that provides a test of the 'data extraction' part of the extraction workflow as described in the resource [data-extraction.md](docs/data-extraction.md). Only check if data extraction can be triggered and that it successfully completes. Disregard all other details. The name of The Function must be 'data_extraction_check'.
+- Implement The Function called "data_extraction_check" (The Data Extraction Check Function) that provides a test of the 'data extraction' part of the extraction workflow as described in the resource [data-extraction.md](docs/data-extraction.md). The Data Extraction Check Function should:
+  - Initialize a single repo "users"
+  - normalize the users data using The Normalization Function
+  - push the normalized users data to The DevRev Servers
+  - Note: The Data Extraction Check Function should be simplified and should disregard all other details.
 
   ***Acceptance Tests:***
 

docs/airdrop_sdk_library_documentation.md

@@ -363,9 +363,8 @@ Defines the structure of events sent to external extractors from Airdrop platfor
   Required. An object containing:
 
   - _secrets_: An object containing:
-    - _service_account_token_: Required. A **string** representing the DevRev authentication token for Airdrop platform
-  - _snap_in_version_id_: Required. A **string** representing the version ID of the snap-in
-  - _snap_in_id_: Required. A **string** representing the snap-in ID.
+    - _service_account_token_: A **string** representing the DevRev authentication token for Airdrop platform
+  - _snap_in_version_id_: A **string** representing the version ID of the snap-in
 
 - _payload_
 
@@ -426,7 +425,7 @@ This function initializes a new worker thread and oversees its lifecycle. It sho
 #### Usage
 
 ```typescript
-spawn({ event, initialState, workerPath, options });
+spawn({ event, initialState, workerPath, options, initialDomainMapping });
 ```
 
 #### Parameters
@@ -443,6 +442,10 @@ spawn({ event, initialState, workerPath, options });
 
   Required. A **string** that represents the path to the worker file.
 
+- _initialDomainMapping_
+
+  Optional. An **object** containing the initial domain mapping.
+
 - _options_
 
   Optional. An object of type **WorkerAdapterOptions**, which will be passed to the newly created worker. This worker will then initialize a `WorkerAdapter` by invoking the `processTask` function. The options include:
@@ -458,10 +461,6 @@ spawn({ event, initialState, workerPath, options });
   - `batchSize`
 
     A **number** that determines the maximum number of items to be processed and saved to an intermediary file before being sent to the Airdrop platform. The default batch size is 2,000.
-  
-  - `initialDomainMapping`
-
-    Optional. A **string** containing the initial domain mapping.
 
 #### Return value
 

docs/data-extraction-rules-from-emitting-events.md

@@ -0,0 +1,8 @@
+Here are the rules when `EXTRACTION_DATA_START` or `EXTRACTION_DATA_CONTINUE` event is received by the snap-in:
+
+- If there's a timeout ("onTimeout" method in "spawn" method), the snap-in should emit an event with type `EXTRACTION_DATA_PROGRESS` (no "data" payload).
+- If the extraction is rate-limited by the external system, the snap-in should emit an event with type `EXTRACTION_DATA_DELAY` ("data" payload should contain attribute "delay" - delay in seconds as an integer).
+- If the extraction is successful and all the resources have been extracted, the snap-in should emit an event with type `EXTRACTION_DATA_DONE` (no "data" payload).
+- In case of unexpected error, the snap-in should emit an event with type `EXTRACTION_DATA_ERROR` ("data" payload should contain attribute "error" - error message as a string).
+
+NOTE: In all cases, only a single event should be emitted.

docs/data-extraction.md

@@ -112,7 +112,6 @@ Behind the scenes, the SDK library stores items pushed to the repository and upl
 ### Data normalization
 
 Extracted data must be normalized to fit the domain metadata defined in the `external-domain-metadata.json` file. 
-More details on this process are provided in the [Metadata extraction](/public/snapin-development/adaas/metadata-extraction) section.
 
 Normalization rules:
 
@@ -124,6 +123,7 @@ Normalization rules:
 - References: references must be strings, not numbers or objects.
 - Number fields must be valid JSON numbers (not strings).
 - Multiselect fields must be provided as an array (not CSV).
+- Rich text: Strings from 3rd party services should be converted to the rich text by splitting the string by newlines, filtering out empty lines, and returning an array of strings.
 
 Extracted items are automatically normalized when pushed to the `repo` if a normalization function is provided under the `normalize` key in the repo object.
 

docs/external-sync-units-extraction.mdx

@@ -0,0 +1,63 @@
+An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev.
+In some systems, this is a project; in some it is a repository; in support systems it could be
+called a brand or an organization.
+What a unit of data is called and what it represents depends on the external system's domain model.
+It usually combines contacts, users, work-like items, and comments into a unit of domain objects.
+
+In the external sync unit extraction phase, the snap-in is expected to obtain a list of external
+sync units that it can extract from the external system API and send it to Airdrop in its response.
+
+External sync unit extraction is executed only during the initial import.
+
+## Triggering event
+
+Airdrop starts the external sync unit extraction by sending a message with the event type `EXTRACTION_EXTERNAL_SYNC_UNITS_START`.
+
+The snap-in must reply to Airdrop with an `EXTRACTION_EXTERNAL_SYNC_UNITS_DONE` message when finished, 
+or `EXTRACTION_EXTERNAL_SYNC_UNITS_ERROR` if an error occurs.
+
+## Implementation
+
+
+The snap-in should emit the list of external sync units in the given format:
+
+```typescript
+const externalSyncUnits: ExternalSyncUnit[] = [
+  {
+    id: "devrev",
+    name: "devrev",
+    description: "Demo external sync unit",
+    item_count: 100,
+  },
+];
+```
+
+- `id`: The unique identifier in the external system.
+- `name`: The human-readable name in the external system.
+- `description`: The short description if the external system provides it.
+- `item_count`: The number of items (issues, tickets, comments or others) in the external system.
+  Item count should be provided if it can be obtained in a lightweight manner, such as by calling an API endpoint.
+  If there is no such way to get it (for example, if the items would need to be extracted to count them),
+  then the item count should be `-1` to avoid blocking the import with long-running queries.
+
+The snap-in must respond to Airdrop with a message, which contains a list of external sync units as a payload:
+
+```typescript
+await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsDone, {
+  external_sync_units: externalSyncUnits,
+});
+```
+
+or an error:
+
+```typescript
+await adapter.emit(ExtractorEventType.ExtractionExternalSyncUnitsError, {
+  error: {
+    message: "Failed to extract external sync units. Lambda timeout.",
+  },
+});
+```
+
+**The snap-in must always emit a single message.**
+
+To test your changes, start a new airdrop in the DevRev App. If external sync units extraction is successful, you should be prompted to choose an external sync unit from the list.

docs/function_invocation.mdx

@@ -1,9 +1,9 @@
-The function is invoked by the runner.ts file.
-
-runner.ts processes events of type "any", which are basically events of type "AirdropEvent" with some additional fields, and passes the events to the function. You can ignore the additional fields.
+A function can be invoked synchronously or asynchronously.
 
 You need to implement the run method in your function. The run method is called when the function is invoked. The run method signature is defined below:
+
 ```typescript
-async function run(events: []AirdropEvent): void;
+async function run(events: []FunctionInput): any;
 ```
-As of now, it's safe to assume that only one event is passed to the function at a time. The function can be invoked multiple times for multiple events.
+
+The value returned from the `run` method is passed back in synchronous execution modes, such as commands, snap kit actions, and event source synchronous execution. In asynchronous execution modes, such as automation execution, the return value is ignored.

docs/incremental_mode.md

@@ -0,0 +1,24 @@
+### Implementing Incremental Data Sync
+
+Incremental data synchronization retrieves only records that have been created or updated since the last successful sync. This process requires persistently tracking the timestamp of the last successful data extraction. On subsequent runs, this timestamp is used to query the source system for changes.
+
+Incremental mode should only be handled if the "event_type" is `EXTRACTION_DATA_START`.
+
+To check if we're in incremental mode, you should check if the value of `adapter.event.payload.event_context.mode` is `SyncMode.INCREMENTAL`.
+
+#### How implement incremental mode
+
+If we're in incremental mode, you should reset The Extraction State, indicating the sync hasn't been completed yet for all data types that we support incremental mode.
+
+Value of `adapter.state.lastSuccessfulSyncStarted` (of format ISO 8601 Extended Format with timezone) represents you the information since when you should query resources from the 3rd party API.
+
+To retrieve only the resources from the API that have to be updated, filtering on The API should be implemented.
+
+Note:
+- `adapter.state.lastSuccessfulSyncStarted` and `adapter.state.lastSyncStarted` are internal properties of the ts-adaas library, so no need to define it. This should be a read-only property.
+
+#### Remember the last successful sync time
+
+If the sync is successful, update the state object with the current time.
+
+Note: No need to modify any time-related properties in adapter.state object.