diff --git a/functions/levante-admin/SYNCHRONOUS_ASSIGNMENT_SYNC.md b/functions/levante-admin/SYNCHRONOUS_ASSIGNMENT_SYNC.md new file mode 100644 index 0000000..0b270f3 --- /dev/null +++ b/functions/levante-admin/SYNCHRONOUS_ASSIGNMENT_SYNC.md @@ -0,0 +1,269 @@ +# Synchronous Assignment Sync + +## Overview + +This document explains the changes made to eliminate the delay between creating/editing an assignment and displaying it on the home page. Previously, assignment creation/updates triggered background Firebase functions that processed assignments asynchronously, causing users to not see their newly created or edited assignments immediately. + +## Problem + +When a user created or edited an assignment via `upsertAdministration`: + +1. The administration document was created/updated in Firestore +2. A Firestore trigger (`syncAssignmentsOnAdministrationUpdate`) was fired asynchronously +3. This trigger processed assignments in the background using task queues +4. Users were redirected to the home page before assignments were visible +5. This created a poor user experience with a noticeable delay + +## Solution + +The solution implements **fully synchronous and atomic assignment processing** within the `upsertAdministration` handler. **All assignment operations are processed synchronously** before the function returns, ensuring immediate visibility of assignments. The system also implements comprehensive atomic rollback - if any participant fails to receive an assignment, the entire operation is rolled back, including the administration document for new administrations. + +## Implementation Details + +### Architecture + +The implementation processes **all org chunks synchronously** within the `upsertAdministration` handler: + +1. **Synchronous Processing**: All org chunks are processed synchronously before the function returns +2. **Background Processing (Redundant)**: The existing Firestore trigger still fires and processes chunks asynchronously, but this is redundant since all chunks are already processed synchronously. The background processing is safe due to idempotent operations but adds unnecessary overhead. + +### Code Structure + +The synchronous sync logic is implemented as two helper functions within `upsertAdministration.ts`: + +- `createAssignments` - Handles new administration creation +- `updateAssignments` - Handles administration updates + +These functions are called directly after the Firestore transaction completes, ensuring assignments are processed before the function returns to the client. + +### Key Functions + +#### `createAssignments` + +Handles synchronous assignment creation for new administrations: + +```typescript +const createAssignments = async ( + administrationId: string, + administrationDocRef: DocumentReference, + currData: IAdministration, + creatorUid?: string, + isNewAdministration: boolean = false +) +``` + +**Process:** + +1. Standardizes administration orgs using `standardizeAdministrationOrgs` +2. Chunks orgs into groups of 100 +3. Processes **all chunks synchronously** using `updateAssignmentsForOrgChunkHandler` +4. Tracks all successfully created assignments in `allCreatedUserIds` for rollback if any chunk fails +5. If any chunk fails, immediately rolls back all previously processed chunks +6. If this is a new administration and any chunk fails, also rolls back the administration document +7. Re-throws errors to ensure the function fails if assignment creation fails + +#### `updateAssignments` + +Handles synchronous assignment updates for modified administrations: + +```typescript +const updateAssignments = async ( + administrationId: string, + administrationDocRef: DocumentReference, + prevData: IAdministration, + currData: IAdministration +) +``` + +**Process:** + +1. Calculates removed orgs by comparing previous and current org lists +2. Removes assignments from users in removed orgs +3. Standardizes administration orgs +4. Processes **all org chunks synchronously** for updates +5. If any chunk fails, throws an error (errors are caught by the handler and logged but not re-thrown for updates) + +### Modified Flow + +**Before:** + +``` +upsertAdministration → Transaction → Return + ↓ + Firestore Trigger (async) → Process Assignments +``` + +**After:** + +``` +upsertAdministration → Transaction → Sync All Assignments (sync, atomic) → Return + ↓ + Firestore Trigger (async) → Process All Chunks (redundant, idempotent) +``` + +The synchronous sync is **fully atomic** - if any participant fails to receive an assignment, the entire operation (including the administration document for new administrations) is rolled back before the function returns. All chunks are processed synchronously, so the background trigger is redundant but safe due to idempotent operations. + +### Implementation Details + +The synchronous sync happens immediately after the transaction commits: + +1. The administration document is fetched from Firestore (to ensure server timestamps are populated) +2. Based on whether it's a create or update operation, the appropriate sync function is called +3. The sync function processes **all org chunks synchronously** +4. The function returns, allowing the client to see all assignments immediately +5. The background trigger still fires but is redundant since all chunks are already processed + +### Integration with Background Trigger + +The background Firestore trigger (`syncAssignmentsOnAdministrationUpdate`) still fires after document writes, but it is **redundant** since all chunks are already processed synchronously: + +- **Idempotent Operations**: All assignment operations are idempotent, so duplicate processing is safe +- **Redundant Processing**: The background trigger processes all chunks again, which is unnecessary overhead +- **Future Optimization**: Consider disabling or skipping the background trigger when all chunks have been processed synchronously to reduce unnecessary processing + +## Configuration + +### Synchronous Processing + +The implementation processes **all org chunks synchronously**, where each chunk contains up to **100 organizations**. This ensures: + +- **Immediate Visibility**: Users see all assignments immediately, regardless of scale +- **Complete Atomicity**: All participants receive assignments or none do (for new administrations) +- **No Partial States**: Eliminates the need for background processing to complete assignments + +**Note**: For very large assignments (hundreds of chunks), this may approach Cloud Functions timeout limits. Monitor function execution times and consider implementing chunk processing limits if timeout issues occur. + +## Transactional Guarantees + +The assignment creation process implements **all-or-nothing semantics** to ensure data integrity: + +### All-or-Nothing Behavior + +When creating assignments for participants, the system guarantees that **all participants receive the assignment, or none do**. This prevents partial assignment states where some participants have assignments while others do not. + +### Rollback Mechanism + +If assignment creation fails for **any** participant in any chunk: + +1. **Automatic Rollback**: The system automatically calls `rollbackAssignmentCreation` to revert all assignment-related changes +2. **Assignment Deletion**: All assignment documents created during the process are deleted +3. **User Document Reversion**: The administration ID is removed from user documents' `assignments` arrays +4. **Statistics Reversion**: Any administration statistics that were incremented are decremented +5. **Administration Document Rollback** (for new administrations only): If this is a new administration, the system also calls `rollbackAdministrationCreation` to: + - Delete the administration document from the `administrations` collection + - Remove the administration ID from the creator's `adminData.administrationsCreated` array +6. **Error Propagation**: The error is thrown, causing the entire `upsertAdministration` operation to fail + +### Rollback Implementation + +The rollback process consists of two functions: + +**`rollbackAssignmentCreation`** (in `assignment-utils.ts`): + +- Processes rollbacks in batches of up to 500 operations for efficiency +- Deletes assignment documents from the `users/{userId}/assignments/{administrationId}` subcollection +- Removes the administration ID from user documents' `assignments.assigned` array +- Deletes the `assignmentsAssigned.{administrationId}` timestamp field +- Decrements administration statistics if org chunk and administration data are provided + +**`rollbackAdministrationCreation`** (in `assignment-utils.ts`): + +- Used only for new administrations when assignment creation fails +- Deletes the administration document from the `administrations` collection +- Removes the administration ID from the creator's `adminData.administrationsCreated` array +- Uses a Firestore transaction to ensure atomicity + +### Chunk-Level Rollback + +The system tracks all successfully created assignments across chunks: + +- Each chunk's successful assignments are tracked in `allCreatedUserIds` +- If any chunk fails, all previously processed chunks are rolled back +- This ensures atomicity across the entire synchronous processing phase + +### Error Handling + +The synchronous sync implements comprehensive error handling: + +- **Chunk-Level Errors**: If `updateAssignmentsForOrgChunkHandler` returns `success: false`, all previous chunks are rolled back and an error is thrown +- **User-Level Errors**: If assignment creation fails for any user within a chunk, that entire chunk is rolled back and an error is thrown +- **Rollback Errors**: If rollback itself fails, errors are logged but the original error is still propagated +- **Final Rollback**: A catch block in `createAssignments` ensures rollback is attempted even if the error occurs outside the chunk loop +- **Error Propagation**: Errors thrown from `createAssignments` are caught by the try-catch block in `upsertAdministrationHandler`. For new administrations, the error is re-thrown to prevent the function from returning successfully. For updates, the error is logged but not re-thrown, allowing the function to return successfully. + +### Error Recovery + +If synchronous sync fails for a **new administration**: + +1. All created assignments are rolled back automatically via `rollbackAssignmentCreation` +2. The administration document is deleted via `rollbackAdministrationCreation` +3. The administration ID is removed from the creator's `adminData.administrationsCreated` array +4. The error is logged with full context (chunk index, user IDs, administration ID, creator UID) +5. The error is re-thrown, causing the `upsertAdministration` function to fail +6. The client receives an error message: "Failed to create assignment for all participants. Please try again." +7. The entire operation is atomic - either everything succeeds or everything is rolled back + +If synchronous sync fails for an **existing administration update**: + +1. All created assignments are rolled back automatically via `rollbackAssignmentCreation` +2. The error is logged but not re-thrown +3. The `upsertAdministration` function returns successfully (the administration document update remains) +4. The background Firestore trigger will process assignments asynchronously as a fallback +5. Users may experience a delay, but assignments will eventually appear via the background trigger + +**Note**: For new administrations, the rollback is complete and atomic - both assignments and the administration document are removed. For updates, only assignments are rolled back since the administration document update is considered successful even if assignment sync fails. + +## Performance Considerations + +### Function Execution Time + +- **All Assignments**: All chunks are processed synchronously before the function returns +- **Execution Time**: Scales linearly with the number of org chunks (each chunk processes up to 100 organizations) +- **Timeout Considerations**: Cloud Functions have a maximum timeout (typically 540 seconds for 2nd gen). For very large assignments, monitor execution time to ensure it stays within limits + +### Transaction Limits + +The implementation respects Firestore transaction limits: + +- Maximum of `MAX_TRANSACTIONS` (100) documents per transaction +- Large user sets are processed in chunks across multiple transactions + +## Testing + +When testing this functionality: + +1. **Verify Immediate Visibility**: Create an assignment and confirm it appears immediately on the home page +2. **Test Large Assignments**: Create assignments with many orgs to verify background processing continues +3. **Test Updates**: Edit existing assignments to verify synchronous updates work correctly +4. **Monitor Logs**: Check for synchronous sync completion logs and background trigger execution + +## Future Improvements + +Potential enhancements: + +1. **Skip Background Processing**: Disable or skip the background trigger when all chunks have been processed synchronously to reduce unnecessary overhead +2. **Progress Tracking**: Add metadata to track which chunks have been processed synchronously (useful if we need to reintroduce chunk limits) +3. **Configurable Chunk Limits**: If timeout issues occur, make chunk processing limits configurable via environment variables +4. **Metrics**: Track synchronous processing performance and identify any timeout issues + +## Related Files + +- `functions/levante-admin/src/upsertAdministration.ts` - Main handler with synchronous sync and rollback logic +- `functions/levante-admin/src/assignments/sync-assignments.ts` - Assignment sync utilities with error handling +- `functions/levante-admin/src/assignments/assignment-utils.ts` - Contains `rollbackAssignmentCreation` and `rollbackAdministrationCreation` functions +- `functions/levante-admin/src/assignments/on-assignment-updates.ts` - Administration stats management +- `functions/levante-admin/src/administrations/sync-administrations.ts` - Background sync handlers +- `functions/levante-admin/src/index.ts` - Firestore trigger definitions + +## Notes + +- The background trigger (`syncAssignmentsOnAdministrationUpdate`) will still fire after document writes, but it is redundant since all chunks are already processed synchronously +- Duplicate processing is safe due to idempotent operations, but adds unnecessary overhead +- The synchronous processing uses the same underlying functions as the background trigger for consistency +- The helper functions (`createAssignments` and `updateAssignments`) are defined as private functions within the `upsertAdministration.ts` file +- For update operations, the previous administration data is fetched before the transaction to enable proper comparison of org changes +- The rollback mechanism ensures **complete atomicity** - if any participant fails to receive an assignment, the entire operation (including the administration document for new administrations) is rolled back +- Rollback happens at the chunk level - if any chunk fails, all previously processed chunks are rolled back +- The `updateAssignmentsForOrgChunkHandler` function also implements its own rollback for failed chunks, providing defense-in-depth +- Statistics are updated synchronously when adding new assignments (mode: "add") to ensure immediate visibility +- **All chunks are processed synchronously** - there is no limit on the number of chunks processed, ensuring complete assignment creation before the function returns diff --git a/functions/levante-admin/src/assignments/assignment-utils.ts b/functions/levante-admin/src/assignments/assignment-utils.ts index bfb444b..c2acd0f 100644 --- a/functions/levante-admin/src/assignments/assignment-utils.ts +++ b/functions/levante-admin/src/assignments/assignment-utils.ts @@ -1,5 +1,10 @@ /* eslint-disable @typescript-eslint/no-non-null-assertion */ -import { getFirestore, Timestamp } from "firebase-admin/firestore"; +import { + getFirestore, + Timestamp, + FieldValue, + FieldPath, +} from "firebase-admin/firestore"; import type { CollectionReference, DocumentData, @@ -30,6 +35,7 @@ import { summarizeAssessmentsForLog, summarizeIdListForLog, } from "../utils/logging.js"; +import { updateAdministrationStatsForOrgChunk } from "./on-assignment-updates.js"; /** * Parse a Firestore Timestamp or Date instance @@ -115,6 +121,151 @@ export const removeAssignmentFromUsers = async ( ); }; +/** + * Rolls back administration document creation by deleting the administration document + * and removing it from the creator's administrationsCreated array. + * This is used when assignment creation fails for a new administration to ensure full atomicity. + * + * @param {string} administrationId - The administration ID to rollback. + * @param {string} creatorUid - The UID of the user who created the administration. + */ +export const rollbackAdministrationCreation = async ( + administrationId: string, + creatorUid: string +) => { + const db = getFirestore(); + + logger.warn(`Rolling back administration document creation`, { + administrationId, + creatorUid, + }); + + try { + const administrationDocRef = db + .collection("administrations") + .doc(administrationId); + + const creatorDocRef = db.collection("users").doc(creatorUid); + + await db.runTransaction(async (transaction) => { + const adminDoc = await transaction.get(administrationDocRef); + const creatorDoc = await transaction.get(creatorDocRef); + + if (adminDoc.exists) { + transaction.delete(administrationDocRef); + } + + if (creatorDoc.exists) { + const fieldPath = new FieldPath("adminData", "administrationsCreated"); + transaction.update( + creatorDocRef, + fieldPath, + FieldValue.arrayRemove(administrationId) + ); + } + }); + + logger.info(`Successfully rolled back administration document creation`, { + administrationId, + creatorUid, + }); + } catch (error: any) { + logger.error("Error rolling back administration document creation", { + error, + administrationId, + creatorUid, + }); + throw error; + } +}; + +/** + * Rolls back assignment creation by deleting assignments and reverting user document updates. + * This is used when assignment creation fails to ensure atomicity. + * + * @param {string[]} userIds - Array of user IDs whose assignments should be rolled back. + * @param {string} administrationId - The administration ID. + * @param {IOrgsList} orgChunk - Optional org chunk to rollback stats for. + * @param {IAdministration} administrationData - Optional administration data for stats rollback. + */ +export const rollbackAssignmentCreation = async ( + userIds: string[], + administrationId: string, + orgChunk?: IOrgsList, + administrationData?: IAdministration +) => { + const db = getFirestore(); + + logger.warn(`Rolling back assignment creation for ${userIds.length} users`, { + administrationId, + userCount: userIds.length, + }); + + const batch = db.batch(); + let batchCount = 0; + const MAX_BATCH_SIZE = 500; + + for (const userId of userIds) { + if (batchCount >= MAX_BATCH_SIZE) { + await batch.commit(); + batchCount = 0; + } + + const assignmentRef = db + .collection("users") + .doc(userId) + .collection("assignments") + .doc(administrationId); + + batch.delete(assignmentRef); + batchCount++; + + const userDocRef = db.collection("users").doc(userId); + const fieldPathDate = new FieldPath( + "assignmentsAssigned", + administrationId + ); + const fieldPathList = new FieldPath("assignments", "assigned"); + + batch.update( + userDocRef, + fieldPathDate, + FieldValue.delete(), + fieldPathList, + FieldValue.arrayRemove(administrationId) + ); + batchCount++; + } + + if (batchCount > 0) { + await batch.commit(); + } + + // Rollback stats if org chunk and administration data are provided + if (orgChunk && administrationData && userIds.length > 0) { + try { + // Decrement stats by the number of users that were rolled back + await updateAdministrationStatsForOrgChunk( + administrationId, + orgChunk, + administrationData, + -userIds.length + ); + } catch (statsError: any) { + logger.error("Error rolling back administration stats", { + statsError, + administrationId, + userCount: userIds.length, + }); + } + } + + logger.info(`Successfully rolled back assignment creation`, { + administrationId, + userCount: userIds.length, + }); +}; + /** * Prepare a new assignment for a user. * @@ -301,6 +452,7 @@ export const addAssignmentToUsers = async ( transaction: Transaction ) => { console.log("hit addAssignmentToUsers"); + const db = getFirestore(); const assignments = await Promise.all( _map(users, (user) => prepareNewAssignment( @@ -312,11 +464,31 @@ export const addAssignmentToUsers = async ( ) ); - return _map(assignments, ([assignmentRef, assignmentData]) => { + return _map(assignments, ([assignmentRef, assignmentData], index: number) => { if (assignmentRef && assignmentData) { logger.debug(`Adding new assignment at ${assignmentRef.path}`, { assignmentSummary: summarizeAssignmentForLog(assignmentData), }); + + const userUid = users[index]; + if (userUid) { + const userDocRef = db.collection("users").doc(userUid); + const fieldPathDate = new FieldPath( + "assignmentsAssigned", + administrationId + ); + const fieldPathList = new FieldPath("assignments", "assigned"); + const dateAssigned = assignmentData.dateAssigned || new Date(); + + transaction.update( + userDocRef, + fieldPathDate, + dateAssigned, + fieldPathList, + FieldValue.arrayUnion(administrationId) + ); + } + return transaction.set(assignmentRef, assignmentData, { merge: true }); } else { return transaction; diff --git a/functions/levante-admin/src/assignments/on-assignment-updates.ts b/functions/levante-admin/src/assignments/on-assignment-updates.ts index 1b0c2d8..0537ca2 100644 --- a/functions/levante-admin/src/assignments/on-assignment-updates.ts +++ b/functions/levante-admin/src/assignments/on-assignment-updates.ts @@ -14,6 +14,7 @@ import type { DocumentDeletedEvent, DocumentUpdatedEvent, } from "../utils/utils.js"; +import type { IAdministration, IOrgsList } from "../interfaces.js"; type Status = "assigned" | "started" | "completed"; @@ -117,6 +118,59 @@ const incrementCompletionStatus = async ( } }; +/** + * Updates administration stats for assignments created for a given org chunk. + * This function is called synchronously to ensure stats are updated immediately. + * + * @param {string} administrationId - The administration ID. + * @param {IOrgsList} orgChunk - The org chunk that was processed. + * @param {IAdministration} administrationData - The administration data. + * @param {number} userCount - The number of users that were assigned (to increment stats by). + */ +export const updateAdministrationStatsForOrgChunk = async ( + administrationId: string, + orgChunk: IOrgsList, + administrationData: IAdministration, + userCount: number +) => { + if (userCount === 0) { + return; + } + + // Allow negative counts for rollback (decrementing stats) + const incrementBy = userCount; + + const db = getFirestore(); + const completionCollectionRef = db + .collection("administrations") + .doc(administrationId) + .collection("stats"); + + const orgList = _reduce( + orgChunk, + (acc: string[], value: string[]) => { + acc.push(...value); + return acc; + }, + [] + ); + orgList.push("total"); + + const taskIds = administrationData.assessments.map((a) => a.taskId); + + await db.runTransaction(async (transaction) => { + await incrementCompletionStatus( + orgList, + "assigned", + taskIds, + completionCollectionRef, + transaction, + userCount, + true + ); + }); +}; + /** * Event handler for when a new assignment document is created. * diff --git a/functions/levante-admin/src/assignments/sync-assignments.ts b/functions/levante-admin/src/assignments/sync-assignments.ts index d9152db..83e0118 100644 --- a/functions/levante-admin/src/assignments/sync-assignments.ts +++ b/functions/levante-admin/src/assignments/sync-assignments.ts @@ -24,11 +24,13 @@ import { getUsersFromOrgs } from "../orgs/org-utils.js"; import { addAssignmentToUsers, updateAssignmentForUsers, + rollbackAssignmentCreation, } from "./assignment-utils.js"; import { summarizeIdListForLog, summarizeOrgsForLog, } from "../utils/logging.js"; +import { updateAdministrationStatsForOrgChunk } from "./on-assignment-updates.js"; /** * Sync globally defined adminstrations with user-specific assignments. @@ -191,85 +193,101 @@ export const updateAssignmentsForOrgChunkHandler = async ({ administrationData: IAdministration; orgChunk: IOrgsList; mode: "update" | "add"; -}) => { +}): Promise<{ success: boolean; userIds: string[]; error?: Error }> => { if (!["update", "add"].includes(mode)) { throw new Error(`Invalid mode: ${mode}. Expected 'update' or 'add'.`); } const db = getFirestore(); + const createdUserIds: string[] = []; - // Get all of the current users and update their assignments. The - // maximum number of docs we can update in a single transaction is - // ``MAX_TRANSACTIONS``. The number of affected users is potentially - // larger. So we loop through chunks of the userIds and update them in - // separate transactions if necessary. - - // ``remainingUsers`` is a placeholder in the event that the number of - // affected users is greater than the maximum number of docs we can update - // in a single transaction. - let remainingUsers: string[] = []; + try { + // Get all of the current users and update their assignments. The + // maximum number of docs we can update in a single transaction is + // ``MAX_TRANSACTIONS``. The number of affected users is potentially + // larger. So we loop through chunks of the userIds and update them in + // separate transactions if necessary. + let usersToUpdate: string[] = []; - // Run the first transaction to get the user list - await db.runTransaction(async (transaction) => { - const usersToUpdate = await getUsersFromOrgs({ - orgs: orgChunk, - transaction, - includeArchived: false, // Do not assign updated assignment to archived users - }); + // Run the first transaction to get the user list + await db.runTransaction(async (transaction) => { + usersToUpdate = await getUsersFromOrgs({ + orgs: orgChunk, + transaction, + includeArchived: false, // Do not assign updated assignment to archived users + }); - logger.info(`Updating assignment ${administrationId} for users`, { - orgChunkSummary: summarizeOrgsForLog(orgChunk), - userSummary: summarizeIdListForLog(usersToUpdate), + logger.info(`Updating assignment ${administrationId} for users`, { + orgChunkSummary: summarizeOrgsForLog(orgChunk), + userSummary: summarizeIdListForLog(usersToUpdate), + }); }); - if (usersToUpdate.length !== 0) { - if (usersToUpdate.length <= MAX_TRANSACTIONS) { - // If the number of users is small enough, update them in this transaction. + const userChunks = _chunk(usersToUpdate, MAX_TRANSACTIONS); + const transactionPromises = userChunks.map((_userChunk) => { + return db.runTransaction(async (transaction) => { if (mode === "update") { return updateAssignmentForUsers( - usersToUpdate, + _userChunk, administrationId, administrationData, transaction ); } else { - console.log("adding assignments to users"); return addAssignmentToUsers( - usersToUpdate, + _userChunk, administrationId, administrationData, transaction ); } - } else { - // Otherwise, just save for the next loop over user chunks. - remainingUsers = usersToUpdate; - return Promise.resolve(usersToUpdate.length); - } - } else { - return Promise.resolve(0); + }); + }); + + await Promise.all(transactionPromises); + + if (mode === "add") { + createdUserIds.push(...usersToUpdate); } - }); - // If remainingUsersToRemove.length === 0, then these chunks will be of zero length - // and the entire loop below is a no-op. - for (const _userChunk of _chunk(remainingUsers, MAX_TRANSACTIONS)) { - await db.runTransaction(async (transaction) => { - if (mode === "update") { - return updateAssignmentForUsers( - _userChunk, + // Update administration stats synchronously for immediate visibility + // Only update stats when adding new assignments (not when updating) + if (mode === "add" && usersToUpdate.length > 0) { + await updateAdministrationStatsForOrgChunk( + administrationId, + orgChunk, + administrationData, + usersToUpdate.length + ); + } + + return { success: true, userIds: createdUserIds }; + } catch (error: any) { + logger.error("Error creating assignments for org chunk", { + error, + administrationId, + orgChunkSummary: summarizeOrgsForLog(orgChunk), + createdUserIds, + }); + + // Rollback all created assignments if any failed + if (mode === "add" && createdUserIds.length > 0) { + try { + await rollbackAssignmentCreation( + createdUserIds, administrationId, - administrationData, - transaction + orgChunk, + administrationData ); - } else { - return addAssignmentToUsers( - _userChunk, + } catch (rollbackError: any) { + logger.error("Error during rollback of assignment creation", { + rollbackError, administrationId, - administrationData, - transaction - ); + userIds: createdUserIds, + }); } - }); + } + + return { success: false, userIds: createdUserIds, error }; } }; diff --git a/functions/levante-admin/src/upsertAdministration.ts b/functions/levante-admin/src/upsertAdministration.ts index 8db727f..16ddcf6 100644 --- a/functions/levante-admin/src/upsertAdministration.ts +++ b/functions/levante-admin/src/upsertAdministration.ts @@ -6,8 +6,29 @@ import { } from "firebase-admin/firestore"; import { HttpsError } from "firebase-functions/v2/https"; import { logger } from "firebase-functions/v2"; -import type { IAssessment, IOrgsList } from "./interfaces.js"; // Assuming necessary types/helpers are in common import type { Class, Group, School } from "../firestore-schema.js"; +import type { IAssessment, IOrgsList, IAdministration } from "./interfaces.js"; // Assuming necessary types/helpers are in common +import { ORG_NAMES } from "./interfaces.js"; +import { standardizeAdministrationOrgs } from "./administrations/administration-utils.js"; +import { updateAssignmentsForOrgChunkHandler } from "./assignments/sync-assignments.js"; +import _reduce from "lodash-es/reduce.js"; +import _pick from "lodash-es/pick.js"; +import _difference from "lodash-es/difference.js"; +import _fromPairs from "lodash-es/fromPairs.js"; +import _map from "lodash-es/map.js"; +import { + chunkOrgs, + getOnlyExistingOrgs, + getExhaustiveOrgs, + getUsersFromOrgs, +} from "./orgs/org-utils.js"; +import { + removeOrgsFromAssignments, + rollbackAssignmentCreation, + rollbackAdministrationCreation, +} from "./assignments/assignment-utils.js"; +import _chunk from "lodash-es/chunk.js"; +import { MAX_TRANSACTIONS } from "./utils/utils.js"; interface UpsertAdministrationData { name: string; @@ -50,6 +71,251 @@ interface IAdministrationDoc { creatorName: string; } +const createAssignments = async ( + administrationId: string, + administrationDocRef: DocumentReference, + currData: IAdministration, + creatorUid?: string, + isNewAdministration: boolean = false +) => { + const { minimalOrgs } = await standardizeAdministrationOrgs({ + administrationId, + administrationDocRef, + currData, + copyToSubCollections: true, + forceCopy: true, + }); + + const orgChunks = chunkOrgs(minimalOrgs, 100); + const allCreatedUserIds: string[] = []; + const processedChunks: Array<{ orgChunk: IOrgsList; userIds: string[] }> = []; + + try { + const chunkResults = await Promise.all( + orgChunks.map(async (orgChunk, i) => { + try { + const result = await updateAssignmentsForOrgChunkHandler({ + administrationId, + administrationData: currData, + orgChunk, + mode: "add", + }); + return { orgChunk, result, chunkIndex: i }; + } catch (error: any) { + const err = error instanceof Error ? error : new Error(String(error)); + return { + orgChunk, + result: { success: false, userIds: [], error: err }, + chunkIndex: i, + }; + } + }) + ); + + for (const { orgChunk, result } of chunkResults) { + if (result.success) { + allCreatedUserIds.push(...result.userIds); + processedChunks.push({ orgChunk, userIds: result.userIds }); + } + } + + const failedChunk = chunkResults.find(({ result }) => !result.success); + if (failedChunk) { + logger.error( + `Assignment creation failed for org chunk ${ + failedChunk.chunkIndex + 1 + }. Rolling back all successful chunks.`, + { + administrationId, + chunkIndex: failedChunk.chunkIndex, + error: failedChunk.result.error, + totalChunksProcessed: processedChunks.length, + } + ); + + // Rollback all successfully created assignments + if (allCreatedUserIds.length > 0) { + await rollbackAssignmentCreation(allCreatedUserIds, administrationId); + } + + // If this is a new administration, also rollback the administration document + if (isNewAdministration && creatorUid) { + await rollbackAdministrationCreation(administrationId, creatorUid); + } + + throw new Error( + `Failed to create assignment for all participants. ${ + failedChunk.result.error?.message || "Unknown error" + }. Please try again.` + ); + } + + logger.info(`Successfully created assignments for all participants`, { + administrationId, + totalUsersAssigned: allCreatedUserIds.length, + chunksProcessed: processedChunks.length, + totalChunks: orgChunks.length, + }); + } catch (error: any) { + // If we get here, either a chunk failed or rollback failed + // Try to rollback any remaining assignments + if (allCreatedUserIds.length > 0) { + try { + await rollbackAssignmentCreation(allCreatedUserIds, administrationId); + } catch (rollbackError: any) { + logger.error("Error during final rollback of assignments", { + rollbackError, + administrationId, + userIds: allCreatedUserIds, + }); + } + } + + // If this is a new administration, also rollback the administration document + if (isNewAdministration && creatorUid) { + try { + await rollbackAdministrationCreation(administrationId, creatorUid); + } catch (adminRollbackError: any) { + logger.error("Error during final rollback of administration document", { + adminRollbackError, + administrationId, + creatorUid, + }); + } + } + + // Re-throw the error so it can be caught by the handler + throw error; + } +}; + +const updateAssignments = async ( + administrationId: string, + administrationDocRef: DocumentReference, + prevData: IAdministration, + currData: IAdministration +) => { + const db = getFirestore(); + const prevOrgs: IOrgsList = _pick(prevData, ORG_NAMES); + const currOrgs: IOrgsList = _pick(currData, ORG_NAMES); + + const removedOrgs = _fromPairs( + _map(Object.entries(currOrgs), ([key, value]) => [ + key, + _difference(prevOrgs[key], value), + ]) + ) as IOrgsList; + + const numRemovedOrgs = _reduce( + removedOrgs, + (sum, value) => (value ? sum + value.length : sum), + 0 + ); + + if (numRemovedOrgs > 0) { + let remainingUsersToRemove: string[] = []; + let removedExhaustiveOrgs: IOrgsList = {}; + + await db.runTransaction(async (transaction) => { + const removedExistingOrgs = await getOnlyExistingOrgs( + removedOrgs, + transaction + ); + removedExhaustiveOrgs = await getExhaustiveOrgs({ + orgs: removedExistingOrgs, + transaction, + includeArchived: true, + }); + const usersToRemove = await getUsersFromOrgs({ + orgs: removedExhaustiveOrgs, + transaction, + includeArchived: true, + }); + + if (usersToRemove.length !== 0) { + if (usersToRemove.length <= MAX_TRANSACTIONS) { + return removeOrgsFromAssignments( + usersToRemove, + [administrationId], + removedExhaustiveOrgs, + transaction + ); + } else { + remainingUsersToRemove = usersToRemove; + return Promise.resolve(usersToRemove.length); + } + } else { + return Promise.resolve(0); + } + }); + + await Promise.all( + _chunk(remainingUsersToRemove, MAX_TRANSACTIONS).map((_userChunk) => + db.runTransaction(async (transaction) => { + return removeOrgsFromAssignments( + _userChunk, + [administrationId], + removedExhaustiveOrgs, + transaction + ); + }) + ) + ); + } + + const { minimalOrgs } = await standardizeAdministrationOrgs({ + administrationId, + administrationDocRef, + currData, + copyToSubCollections: true, + forceCopy: true, + }); + + const orgChunks = chunkOrgs(minimalOrgs, 100); + + const chunkResults = await Promise.all( + orgChunks.map(async (orgChunk, i) => { + try { + const result = await updateAssignmentsForOrgChunkHandler({ + administrationId, + administrationData: currData, + orgChunk, + mode: "update", + }); + return { result, chunkIndex: i }; + } catch (error: any) { + const err = error instanceof Error ? error : new Error(String(error)); + return { + result: { success: false, userIds: [], error: err }, + chunkIndex: i, + }; + } + }) + ); + + const failedChunk = chunkResults.find(({ result }) => !result.success); + if (failedChunk) { + logger.error( + `Assignment update failed for org chunk ${failedChunk.chunkIndex + 1}.`, + { + administrationId, + chunkIndex: failedChunk.chunkIndex, + error: failedChunk.result.error, + } + ); + throw new Error( + `Failed to update assignment for all participants. ${ + failedChunk.result.error?.message || "Unknown error" + }. Please try again.` + ); + } + + logger.info(`Successfully updated assignments for all participants`, { + administrationId, + chunksProcessed: orgChunks.length, + }); +}; + export const upsertAdministrationHandler = async ( callerAdminUid: string, data: UpsertAdministrationData @@ -180,12 +446,22 @@ export const upsertAdministrationHandler = async ( // 5. Firestore Transaction try { + let prevData: IAdministration | undefined; + + if (administrationId) { + const prevDoc = await db + .collection("administrations") + .doc(administrationId) + .get(); + if (prevDoc.exists) { + prevData = prevDoc.data() as IAdministration; + } + } + const newAdministrationId = await db.runTransaction(async (transaction) => { let administrationDocRef: DocumentReference; - let operationType: string; // To log 'create' or 'update' if (administrationId) { - operationType = "update"; administrationDocRef = db .collection("administrations") .doc(administrationId); @@ -239,7 +515,6 @@ export const upsertAdministrationHandler = async ( transaction.update(administrationDocRef, updateData); // Switched from set with merge to update } else { // --- Create Path --- - operationType = "create"; administrationDocRef = db.collection("administrations").doc(); // --- Read 1 (Create Path) --- Check if user doc exists BEFORE any writes @@ -363,8 +638,9 @@ export const upsertAdministrationHandler = async ( ); } } - logger.info(`Successfully prepared administration ${operationType}`, { + logger.info("Successfully prepared administration", { administrationId: administrationDocRef.id, + operationType: administrationId ? "update" : "create", }); return administrationDocRef.id; // Return the ID }); // End Transaction @@ -372,6 +648,88 @@ export const upsertAdministrationHandler = async ( logger.info("Finished administration upsert transaction", { administrationId: newAdministrationId, }); + + // Sync assignments synchronously for immediate visibility + try { + const administrationDocRef = db + .collection("administrations") + .doc(newAdministrationId); + const administrationDoc = await administrationDocRef.get(); + + if (!administrationDoc.exists) { + logger.warn( + `Administration ${newAdministrationId} not found after creation. Skipping sync.` + ); + return { status: "ok", administrationId: newAdministrationId }; + } + + const administrationData = administrationDoc.data() as IAdministration; + const isNewAdministration = !administrationId; + const creatorUid = administrationData?.createdBy; + + if (isNewAdministration) { + if (!creatorUid) { + logger.error( + `Cannot sync assignments: administration ${newAdministrationId} has no createdBy field`, + { administrationId: newAdministrationId } + ); + throw new HttpsError( + "internal", + "Administration document is missing creator information" + ); + } + await createAssignments( + newAdministrationId, + administrationDocRef, + administrationData, + creatorUid, + true + ); + } else if (prevData) { + await updateAssignments( + newAdministrationId, + administrationDocRef, + prevData, + administrationData + ); + } else { + if (!creatorUid) { + logger.error( + `Cannot sync assignments: administration ${newAdministrationId} has no createdBy field`, + { administrationId: newAdministrationId } + ); + throw new HttpsError( + "internal", + "Administration document is missing creator information" + ); + } + await createAssignments( + newAdministrationId, + administrationDocRef, + administrationData, + creatorUid, + true + ); + } + + logger.info("Finished synchronous assignment sync", { + administrationId: newAdministrationId, + }); + } catch (syncError: any) { + logger.error("Error during synchronous assignment sync", { + error: syncError, + administrationId: newAdministrationId, + }); + + // If this is a new administration and sync failed, the error was already thrown + // and should have triggered rollback. Re-throw to prevent the function from + // returning successfully when assignments failed to be created. + const isNewAdministration = !administrationId; + if (isNewAdministration) { + throw syncError; + } + } + return { status: "ok", administrationId: newAdministrationId }; } catch (error: any) { logger.error("Error during administration upsert", { error }); diff --git a/package.json b/package.json index e9575ad..e6265bf 100644 --- a/package.json +++ b/package.json @@ -8,6 +8,7 @@ ], "scripts": { "build": "npm run build --workspaces", + "build:watch": "npm run build:watch --workspaces", "format": "npm run format --workspaces", "lint": "npm run lint --workspaces", "prepare": "husky install || true",