feat: URL store data model and data access #1050
Open
+1,490
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
packages/spacecat-shared-data-access/src/models/audit-url/audit-url.schema.js
Fixed
Show fixed
Hide fixed
- Add rank and traffic fields to AuditUrl schema (optional, nullable) - Implement sortAuditUrls static method for sorting by multiple fields - Add allBySiteIdSorted and allBySiteIdAndSourceSorted methods - Update allBySiteIdAndAuditType to support sorting - Add TypeScript definitions for new fields and methods - Add 20 comprehensive unit tests for sorting functionality - All tests passing (1147 total) - Code coverage: 98.01% for audit-url module
- Add NODE_OPTIONS with --max-old-space-size=4096 to prevent heap out of memory errors - Fixes FATAL ERROR: JavaScript heap out of memory in CI pipeline
Add platformType field to AuditUrl schema to categorize URLs as primary-site or offsite platforms (Wikipedia, YouTube, social media, etc.). Changes: - Add platformType attribute with 11 supported platform types - Add GSI for efficient querying by siteId and platformType - Add collection methods: allBySiteIdAndPlatform(), allOffsiteUrls() - Add model helper methods: isOffsitePlatform(), isPlatformType() - Export PLATFORM_TYPES constant - Update TypeScript definitions - Add 33 comprehensive unit tests Platform types supported: primary-site, wikipedia, youtube-channel, reddit-community, facebook-page, twitter-profile, linkedin-company, instagram-account, tiktok-account, github-org, medium-publication All methods support sorting and pagination.
nitinja
approved these changes
Nov 18, 2025
Member
nitinja
left a comment
nitinja
left a comment
There was a problem hiding this comment.
LGTM, cant find any issues. Modern JS optional chaining can be used to shorten some of the functions like
this.getPlatformType?.() ?? this.platformType
Member
|
I briefly discussed this with @ravkiran |
This reverts commit 483b27a.
Replace the 'source' string attribute with 'byCustomer' boolean to simplify the data model and clearly distinguish between customer-added (true) and system-added (false) URLs. Changes: - Schema: Replace 'source' with 'byCustomer' (boolean, default: true) - Model: Replace isManualSource() with isCustomerUrl() - Collection: Replace allBySiteIdAndSource with allBySiteIdByCustomer - Collection: Replace allBySiteIdAndSourceSorted with allBySiteIdByCustomerSorted - Collection: Replace removeForSiteIdAndSource with removeForSiteIdByCustomer - Update GSI index from siteId+source to siteId+byCustomer - Update TypeScript definitions Migration mapping: - source='manual' → byCustomer=true - source='sitemap'/'discovery'/other → byCustomer=false
- Update test fixtures to use byCustomer instead of source - Update model tests to use isCustomerUrl instead of isManualSource - Update collection tests for byCustomer methods
- Update all unit tests for model and collection - Update all integration tests - Replace source with byCustomer throughout - Replace isManualSource with isCustomerUrl - Replace allBySiteIdAndSource with allBySiteIdByCustomer - Replace removeForSiteIdAndSource with removeForSiteIdByCustomer
Resolve conflict in test/it/util/db.js: - Use main branch's approach for disabling debug logging - Keep console as default logger and disable debug inline - Both approaches achieve same goal of preventing memory issues
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Please ensure your pull request adheres to the following guidelines:
Related Issues
Thanks for contributing!