diff --git a/README.md b/README.md index facd0c371..375e23bdb 100644 --- a/README.md +++ b/README.md @@ -321,9 +321,12 @@ For the complete version, please refer to [this example](./examples/12-transform #### `deriveLinkedEntries(config)` -For each entry of the given content type (source entry), derives a new entry and sets up a reference to it on the source entry. The content of the new entry is generated by the user-provided `deriveEntryForLocale` function. -For each source entry, this function will be called as many times as there are locales in the space. Each time, it will be called with the `from` fields and one of the locales as arguments. -The derive function is expected to return an object with the desired target fields. If it returns `undefined`, the new entry will have no values for the current locale. +For each entry of the given content type (source entry), derives linked entries and sets references on the source entry. + +There are two modes: + +- Single-entry mode (existing): provide `deriveEntryForLocale` to derive a single entry and write a single link (or a single-element array if the destination is an Array). +- Multi-entry mode (new): provide `deriveEntriesForLocale` to derive 0..N entries per locale and write an array of entry links. **`config : Object`** – Entry derivation definition, with the following properties: @@ -341,7 +344,7 @@ The derive function is expected to return an object with the desired target fiel - `fields` is an object containing each of the `from` fields. Each field will contain their current localized values (i.e. `fields == {myField: {'en-US': 'my field value'}}`) -- **`deriveEntryForLocale : function (fields, locale, {id}): object`** _(required)_ – Function that generates the field values for the derived entry. +- **`deriveEntryForLocale : function (fields, locale, {id}): object`** _(required for single-entry derivation)_ – Function that generates the field values for the derived entry. - `fields` is an object containing each of the `from` fields. Each field will contain their current localized values (i.e. `fields == {myField: {'en-US': 'my field value'}}`) - `locale` one of the locales in the space being transformed @@ -349,6 +352,12 @@ The derive function is expected to return an object with the desired target fiel The return value must be an object with the same keys as specified in `derivedFields`. Their values will be written to the respective new entry fields for the current locale (i.e. `{nameField: 'myNewValue'}`) +- **`deriveEntriesForLocale : function (fields, locale, {id}): { fields }[]`** _(required for multi-entry derivation)_ – Return an array of child entries to upsert for the current locale. Each array element must be shaped like CMA entries: `{ fields: { fieldId: { [locale]: value } } }`. + +- **`derivedEntryId: function ({ sourceId, locale, index, candidate }): string`** _(optional)_ – Custom deterministic ID generator for each derived child in multi-entry mode. Defaults to `${sourceId}-${toReferenceField}-${locale}-${index}` (truncated to 64 chars with a short hash suffix if needed). + +- **`publishDerived : 'always' | 'never' | 'preserve'`** _(optional)_ – Publish behavior for derived child entries in multi-entry mode. Defaults to `'preserve'`. Parent publish behavior is still controlled by `shouldPublish`. + - **`shouldPublish : bool|'preserve'`** _(optional)_ – If true, both the source and the derived entries will be published. If false, both will remain in draft state. If preserve, will keep current states of the source entries (default `true`) ##### `deriveLinkedEntries(config)` Example @@ -379,6 +388,61 @@ migration.deriveLinkedEntries({ For the complete version of this migration, please refer to [this example](./examples/15-derive-entry-n-to-1.js). +##### Deriving multiple entries (multi-entry mode) + +When deriving multiple entries per locale, the destination field `toReferenceField` must be an `Array>` and, if it has `linkContentType` validation, it must include `derivedContentType`. + +```javascript +migration.deriveLinkedEntries({ + contentType: 'post', + from: ['raw'], + toReferenceField: 'items', + derivedContentType: 'jsonItem', + derivedFields: ['kind', 'key', 'index', 'payload'], + shouldPublish: 'preserve', + // Multi-entry derivation: return 0..N child entries for each (source, locale) + deriveEntriesForLocale: (from, locale, { id }) => { + const raw = from.raw?.[locale] + if (raw == null) return [] + if (Array.isArray(raw)) { + return raw.map((value, i) => ({ + fields: { + kind: { [locale]: 'array' }, + index: { [locale]: i }, + payload: { [locale]: value } + } + })) + } + if (typeof raw === 'object') { + return Object.entries(raw).map(([k, v]) => ({ + fields: { + kind: { [locale]: 'object' }, + key: { [locale]: String(k) }, + payload: { [locale]: v } + } + })) + } + return [ + { + fields: { + kind: { [locale]: 'scalar' }, + index: { [locale]: 0 }, + payload: { [locale]: raw } + } + } + ] + }, + // Optional custom child IDs; defaults to `${sourceId}-${toReferenceField}-${locale}-${index}` + derivedEntryId: ({ sourceId, locale, index }) => + `${sourceId}-items-${locale}-${index}`.slice(0, 63) +}) +``` + +Notes: + +- Only locales present in the source `from` fields are written to children. +- Re-runs are idempotent: children are upserted by deterministic IDs and links are set positionally per locale. An empty array result writes `[]` to the destination field. + #### `transformEntriesToType(config)` For the given (source) content type, transforms all its entries according to the user-provided `transformEntryForLocale` function into a new entry of a specific different (target) content type. For each entry, the CLI will call the function `transformEntryForLocale` once per locale in the space, passing in the `from` fields and the locale as arguments. The transform function is expected to return an object with the desired target fields. If it returns `undefined`, this entry locale will be left untouched. diff --git a/index.d.ts b/index.d.ts index f7e281f68..8027464db 100644 --- a/index.d.ts +++ b/index.d.ts @@ -598,13 +598,33 @@ export interface IDeriveLinkedEntriesConfig { /** (optional) – If true, both the source and the derived entries will be published. If false, both will remain in draft state. If preserve, will keep current states of the source entries (default true) */ shouldPublish?: boolean | 'preserve' /** - * (required) – Function that generates the field values for the derived entry. + * (required for single-entry derivation) – Function that generates the field values for the derived entry. * fields is an object containing each of the from fields. Each field will contain their current localized values (i.e. fields == {myField: {'en-US': 'my field value'}}) * locale one of the locales in the space being transformed * * The return value must be an object with the same keys as specified in derivedFields. Their values will be written to the respective new entry fields for the current locale (i.e. {nameField: 'myNewValue'}) */ - deriveEntryForLocale: (inputFields: ContentFields, locale: string) => { [field: string]: any } + deriveEntryForLocale?: (inputFields: ContentFields, locale: string) => { [field: string]: any } + + /** (required for multi-entry derivation) – Function that generates 0..N child entries for this (source, locale). If provided, overrides deriveEntryForLocale. */ + deriveEntriesForLocale?: ( + fromFields: ContentFields, + locale: string, + context: { id: string } + ) => + | Array<{ fields: { [field: string]: { [locale: string]: any } } }> + | Promise> + + /** (optional) – control deterministic IDs per derived child; if omitted, default is `${sourceId}-${toReferenceField}-${locale}-${index}`. */ + derivedEntryId?: (params: { + sourceId: string + locale: string + index: number + candidate: { fields: { [field: string]: { [locale: string]: any } } } + }) => string + + /** (optional) – publish behavior for derived entries; default 'preserve' */ + publishDerived?: 'always' | 'never' | 'preserve' } type TagVisibility = 'private' | 'public' diff --git a/src/lib/action/entry-derive.ts b/src/lib/action/entry-derive.ts index 4ceed67f8..67616068e 100644 --- a/src/lib/action/entry-derive.ts +++ b/src/lib/action/entry-derive.ts @@ -7,20 +7,34 @@ import isDefined from '../utils/is-defined' import Entry from '../entities/entry' import * as _ from 'lodash' import shouldPublishLocalChanges from '../utils/should-publish-local-changes' +import { createHash } from 'crypto' class EntryDeriveAction extends APIAction { private contentTypeId: string private fromFields: string[] private referenceField: string private derivedContentType: string + private derivedFields: string[] private deriveEntryForLocale: ( inputFields: any, locale: string, { id }: { id: string } ) => Promise + private deriveEntriesForLocale?: ( + inputFields: any, + locale: string, + context: { id: string } + ) => Promise }>> | Array<{ fields: Record }> private identityKey: (fromFields: any) => Promise private shouldPublish: boolean | 'preserve' private useLocaleBasedPublishing: boolean + private derivedEntryId?: (params: { + sourceId: string + locale: string + index: number + candidate: { fields: Record } + }) => string + private publishDerived?: 'always' | 'never' | 'preserve' constructor(contentTypeId: string, entryDerivation: EntryDerive) { super() @@ -28,7 +42,9 @@ class EntryDeriveAction extends APIAction { this.fromFields = entryDerivation.from this.referenceField = entryDerivation.toReferenceField this.derivedContentType = entryDerivation.derivedContentType + this.derivedFields = entryDerivation.derivedFields this.deriveEntryForLocale = entryDerivation.deriveEntryForLocale + this.deriveEntriesForLocale = entryDerivation.deriveEntriesForLocale this.identityKey = entryDerivation.identityKey this.shouldPublish = isDefined(entryDerivation.shouldPublish) ? entryDerivation.shouldPublish @@ -36,6 +52,8 @@ class EntryDeriveAction extends APIAction { this.useLocaleBasedPublishing = isDefined(entryDerivation.useLocaleBasedPublishing) ? entryDerivation.useLocaleBasedPublishing : false + this.derivedEntryId = entryDerivation.derivedEntryId + this.publishDerived = entryDerivation.publishDerived } private async publishEntry(api: OfflineAPI, entry: Entry, locales: string[]) { @@ -46,6 +64,17 @@ class EntryDeriveAction extends APIAction { } } + private isDerivedFieldAllowed(fieldId: string): boolean { + return this.derivedFields?.includes(fieldId) + } + + private resolveChildPublishMode(): boolean | 'preserve' { + if (this.publishDerived === 'always') return true + if (this.publishDerived === 'never') return false + if (this.publishDerived === 'preserve') return 'preserve' + return this.shouldPublish + } + async applyTo(api: OfflineAPI) { const entries: Entry[] = await api.getEntriesForContentType(this.contentTypeId) const locales: string[] = await api.getLocalesForSpace() @@ -56,6 +85,132 @@ class EntryDeriveAction extends APIAction { const newEntryId = await this.identityKey(inputs) const hasEntry = await api.hasEntry(newEntryId) + // Multi-entry mode: derive multiple children and set array of links + if (this.deriveEntriesForLocale) { + const field = sourceContentType.fields.getField(this.referenceField) + // Validate destination is Array> + if ( + !( + field && + field.type === 'Array' && + field.items?.type === 'Link' && + field.items?.linkType === 'Entry' + ) + ) { + await api.recordRuntimeError( + new Error( + `Invalid destination: field "${this.referenceField}" must be of type Array> to use deriveEntriesForLocale.` + ) + ) + continue + } + + // If validations specify linkContentType ensure includes derivedContentType + const linkContentTypeValidation = (field.items?.validations || []).find((v) => + Array.isArray(v?.linkContentType) + ) + if ( + linkContentTypeValidation && + !linkContentTypeValidation.linkContentType.includes(this.derivedContentType) + ) { + await api.recordRuntimeError( + new Error( + `Destination field "${this.referenceField}" validations.linkContentType does not include "${this.derivedContentType}".` + ) + ) + continue + } + + // For each locale, derive child entries and link them positionally + for (const locale of locales) { + let derivedList: Array<{ fields: Record }> = [] + try { + const res = await this.deriveEntriesForLocale(inputs, locale, { id: entry.id }) + derivedList = Array.isArray(res) ? res : [] + } catch (err) { + await api.recordRuntimeError(err) + continue + } + + const childIdsInOrder: string[] = [] + for (let index = 0; index < derivedList.length; index++) { + const candidate = derivedList[index] + const targetId = this.derivedEntryId + ? this.derivedEntryId({ sourceId: entry.id, locale, index, candidate }) + : defaultDerivedChildId(entry.id, this.referenceField, locale, index) + + childIdsInOrder.push(targetId) + + const childExists = await api.hasEntry(targetId) + if (!childExists) { + const targetEntry = await api.createEntry(this.derivedContentType, targetId) + // Only write declared derivedFields for this locale + for (const [fieldId, localizedField] of _.entries(candidate.fields)) { + if (!this.isDerivedFieldAllowed(fieldId)) continue + if (!targetEntry.fields[fieldId]) { + targetEntry.setField(fieldId, {}) + } + const value = _.get(localizedField, locale) + if (value !== undefined) { + targetEntry.setFieldForLocale(fieldId, locale, value) + } + } + await api.saveEntry(targetEntry.id) + // Publish behavior for derived children + const childPublishMode = this.resolveChildPublishMode() + if ( + shouldPublishLocalChanges(childPublishMode, entry, this.useLocaleBasedPublishing) + ) { + if (this.useLocaleBasedPublishing) { + await api.localeBasedPublishEntry(targetEntry.id, [locale]) + } else { + await api.publishEntry(targetEntry.id) + } + } + } else { + // Update existing entry fields for this locale (upsert semantics) + const targetEntry = ( + await api.getEntriesForContentType(this.derivedContentType) + ).find((e) => e.id === targetId) + for (const [fieldId, localizedField] of _.entries(candidate.fields)) { + if (!this.isDerivedFieldAllowed(fieldId)) continue + if (!targetEntry.fields[fieldId]) { + targetEntry.setField(fieldId, {}) + } + const value = _.get(localizedField, locale) + if (value !== undefined) { + targetEntry.setFieldForLocale(fieldId, locale, value) + } + } + await api.saveEntry(targetEntry.id) + const childPublishMode = this.resolveChildPublishMode() + if ( + shouldPublishLocalChanges(childPublishMode, entry, this.useLocaleBasedPublishing) + ) { + if (this.useLocaleBasedPublishing) { + await api.localeBasedPublishEntry(targetEntry.id, [locale]) + } else { + await api.publishEntry(targetEntry.id) + } + } + } + } + + // Write array of links for this locale (can be empty) + const links = childIdsInOrder.map((id) => ({ + sys: { type: 'Link', linkType: 'Entry', id } + })) + entry.setFieldForLocale(this.referenceField, locale, links) + } + + await api.saveEntry(entry.id) + if (shouldPublishLocalChanges(this.shouldPublish, entry, this.useLocaleBasedPublishing)) { + await this.publishEntry(api, entry, locales) + } + continue + } + + // Single-entry mode let skipEntry = true let fieldsForTargetEntry = {} @@ -138,3 +293,22 @@ class EntryDeriveAction extends APIAction { } export { EntryDeriveAction } + +function defaultDerivedChildId( + sourceId: string, + toReferenceField: string, + locale: string, + index: number +): string { + const base = `${sourceId}-${toReferenceField}-${locale}-${index}` + if (base.length <= 64) return base + const hash = shortHash(base) + const maxBase = 64 - (1 + hash.length) + return `${base.slice(0, Math.max(0, maxBase))}-${hash}` +} + +function shortHash(input: string): string { + const hashHex = createHash('sha256').update(input).digest('hex') + const base36 = BigInt('0x' + hashHex).toString(36) + return base36.slice(0, 6) +} diff --git a/src/lib/intent-validator/entry-derive.ts b/src/lib/intent-validator/entry-derive.ts index 26cdf4667..67745df77 100644 --- a/src/lib/intent-validator/entry-derive.ts +++ b/src/lib/intent-validator/entry-derive.ts @@ -1,6 +1,7 @@ import Intent from '../intent/base-intent' import SchemaValidator from './schema-validator' import * as Joi from 'joi' +import ValidationError from '../interfaces/errors' class EntryDeriveIntentValidator extends SchemaValidator { protected article = 'an' @@ -26,10 +27,44 @@ class EntryDeriveIntentValidator extends SchemaValidator { derivedFields: Joi.array().items(Joi.string()).required(), identityKey: Joi.func().required(), shouldPublish: Joi.alternatives().try(Joi.boolean(), Joi.string().valid('preserve')), - deriveEntryForLocale: Joi.func().required(), + // exactly one of deriveEntryForLocale or deriveEntriesForLocale must be provided + deriveEntryForLocale: Joi.func(), + deriveEntriesForLocale: Joi.func(), + derivedEntryId: Joi.func(), + publishDerived: Joi.string().valid('always', 'never', 'preserve'), useLocaleBasedPublishing: Joi.boolean().optional() } } + + validate(intent: Intent): ValidationError[] { + const baseErrors = super.validate(intent) + if (baseErrors.length) return baseErrors + + const derivation = intent.toRaw().payload[this.propertyNameToValidate] + const message = + 'Either "deriveEntryForLocale" or "deriveEntriesForLocale" must be provided for entry derivation.' + const presenceSchema = Joi.alternatives() + .try( + Joi.object({ deriveEntryForLocale: Joi.func().required() }).unknown(true), + Joi.object({ deriveEntriesForLocale: Joi.func().required() }).unknown(true) + ) + .messages({ + 'alternatives.match': message + }) + + const { error } = presenceSchema.validate(derivation) + if (error) { + return baseErrors.concat([ + { + type: 'InvalidType', + message, + details: { intent } + } + ]) + } + + return baseErrors + } } export default EntryDeriveIntentValidator diff --git a/src/lib/interfaces/entry-derive.ts b/src/lib/interfaces/entry-derive.ts index e3f2f1268..3fd52d0cd 100644 --- a/src/lib/interfaces/entry-derive.ts +++ b/src/lib/interfaces/entry-derive.ts @@ -6,5 +6,36 @@ export default interface EntryDerive { identityKey: (fromFields: any) => Promise shouldPublish?: boolean | 'preserve' useLocaleBasedPublishing?: boolean - deriveEntryForLocale(inputFields: any, locale: string): Promise + /** + * Single-entry mode: generate field values for a single derived entry for a given locale. + * If multi-entry mode is used (deriveEntriesForLocale), this function is ignored. + */ + deriveEntryForLocale?(inputFields: any, locale: string): Promise + + /** + * Multi-entry mode: generate zero or more derived child entries for a given locale. + * If provided, overrides deriveEntryForLocale. + */ + deriveEntriesForLocale?: ( + inputFields: any, + locale: string, + context: { id: string } + ) => Promise }>> | Array<{ fields: Record }> + + /** + * Optional custom ID generator for each derived child entry. + * Defaults to `${sourceId}-${toReferenceField}-${locale}-${index}` (truncated to 64 chars with a short hash suffix if needed). + */ + derivedEntryId?: (params: { + sourceId: string + locale: string + index: number + candidate: { fields: Record } + }) => string + + /** + * Optional publish behavior for derived child entries only. + * If omitted, falls back to shouldPublish. + */ + publishDerived?: 'always' | 'never' | 'preserve' } diff --git a/test/unit/lib/actions/entry-derive.spec.js b/test/unit/lib/actions/entry-derive.spec.js index a09df5bee..ab5b61929 100644 --- a/test/unit/lib/actions/entry-derive.spec.js +++ b/test/unit/lib/actions/entry-derive.spec.js @@ -365,16 +365,16 @@ describe('Entry Derive', function () { return fromFields.owner['en-US'].toLowerCase().replace(' ', '-') }, shouldPublish: true, - deriveEntryForLocale: async (inputFields, locale) => { + // Multi-entry mode: two children out of a two-word owner + deriveEntriesForLocale: async (inputFields, locale) => { if (locale !== 'en-US') { - return + return [] } const [firstName, lastName] = inputFields.owner[locale].split(' ') - - return { - firstName, - lastName - } + return [ + { fields: { firstName: { [locale]: firstName }, lastName: { [locale]: '' } } }, + { fields: { firstName: { [locale]: '' }, lastName: { [locale]: lastName } } } + ] } }) @@ -427,16 +427,74 @@ describe('Entry Derive', function () { api.stopRecordingRequests() const batches = await api.getRequestBatches() - expect(batches[0].requests.length).to.eq(4) - const createTargetEntryFields = batches[0].requests[0].data.fields - const updateEntryWithLinkFields = batches[0].requests[2].data.fields - expect(createTargetEntryFields.firstName['en-US']).to.eq('johnny') // target entry has first and last name - expect(createTargetEntryFields.lastName['en-US']).to.eq('depp') - expect(typeof updateEntryWithLinkFields.owners['en-US'][0].sys).to.eq('object') // request to update entry is n to n link + // Requests: create first child, maybe publish, create second child, maybe publish, update parent, maybe publish + expect( + batches[0].requests.some( + (r) => r.url.startsWith('/entries/') && !r.url.endsWith('/published') + ) + ).to.eq(true) + const updateEntryWithLinkFields = + batches[0].requests[batches[0].requests.length - 2].data.fields + expect(Array.isArray(updateEntryWithLinkFields.owners['en-US'])).to.eq(true) + expect(updateEntryWithLinkFields.owners['en-US'].length).to.eq(2) expect(updateEntryWithLinkFields.owners['en-US'][0].sys.type).to.eq('Link') - expect(updateEntryWithLinkFields.owners['en-US'][0].sys.id).to.eq( - batches[0].requests[0].data.sys.id - ) // id of linked object is same as id of target object + }) + + it('multi-entry empty list writes empty array', async function () { + const action = new EntryDeriveAction('post', { + derivedContentType: 'child', + from: ['raw'], + toReferenceField: 'items', + derivedFields: ['payload'], + identityKey: async () => 'unused', + shouldPublish: false, + deriveEntriesForLocale: async (from, locale) => { + const value = from.raw?.[locale] + if (value == null) return [] + return [] + } + }) + + const contentTypes = new Map() + contentTypes.set( + 'post', + new ContentType({ + sys: { id: 'post' }, + fields: [ + { + name: 'items', + id: 'items', + type: 'Array', + items: { + type: 'Link', + linkType: 'Entry', + validations: [{ linkContentType: ['child'] }] + } + }, + { id: 'raw', name: 'raw', type: 'Symbol', localized: true } + ] + }) + ) + + const entries = [ + new Entry( + makeApiEntry({ + id: '1', + contentTypeId: 'post', + version: 1, + fields: { raw: { 'en-US': null } } + }) + ) + ] + + const api = new OfflineApi({ contentTypes, entries, locales: ['en-US'] }) + api.startRecordingRequests(null) + await action.applyTo(api) + api.stopRecordingRequests() + const batches = await api.getRequestBatches() + + const updateReq = batches[0].requests.find((r) => r.url === '/entries/1') + expect(updateReq.data.fields.items['en-US']).to.eql([]) }) it('provides entry id', async function () { diff --git a/test/unit/lib/intent-validator/entry-derive.spec.ts b/test/unit/lib/intent-validator/entry-derive.spec.ts index 595abbdd4..1a4814a0e 100644 --- a/test/unit/lib/intent-validator/entry-derive.spec.ts +++ b/test/unit/lib/intent-validator/entry-derive.spec.ts @@ -37,6 +37,24 @@ describe('Entry derivation', function () { expect(validationErrors).to.eql([]) }) + + it('returns no validation errors for multi-entry mode', async function () { + const validationErrors = await validateSteps(function up(migration) { + migration.deriveLinkedEntries({ + contentType: 'post', + derivedContentType: 'child', + from: ['raw'], + toReferenceField: 'items', + derivedFields: ['payload'], + identityKey: async () => 'ignored', + shouldPublish: false, + deriveEntriesForLocale: (from, locale) => { + return [] + } + }) + }) + expect(validationErrors).to.eql([]) + }) }) describe('when using the wrong type for the properties', function () {