Basic integration guide

.github/ISSUE_TEMPLATE/new-artist-integration-evaluation.md

@@ -0,0 +1,44 @@
+---
+name: New artist integration evaluation
+about: Form for evaluating the integration of a new artist into the Spindexer
+title: "[NEW ARTIST INTEGRATION]"
+labels: ''
+assignees: ''
+
+---
+
+### Motivation
+
+When a request for new artist integration into the Spindexer is made we need to technically evaluate the request upfront.
+
+A developer should read the [basic integration guide](/docs/basic_integration_guide.md) and collect the information required for the "Exploring and Collecting Details" section.
+
+### Checklist for evaluation of integration request
+
+```ts
+// Use a `platform` to represent the artist or artist collaboration.
+const EXAMPLE_PLATFORM: MusicPlatform = {
+  id: '0x719...',                              // artist/artist-collaboration address [1]
+  type: MusicPlatformType['x-track-y-prints'], // # of tracks & # of NFTs minted      [2]
+  name: 'danielallan.xyz',                     // artist's official website           [3]
+}
+
+// Use an `nftFactory` to represent the contract which houses the album/collection/tracks.
+const EXAMPLE_ALBUM_OR_TRACK_COLLECTION: NftFactory = {
+{
+  id: '0x141...',                            // album/collection/track address              [4]
+  startingBlock: '15151004',                 // block number before contract inception      [5]
+  platformId: EXAMPLE_PLATFORM.id,           // new/existing platform id                    [6]
+  standard: NFTStandard.ERC721,              // contract type (relates to the chain)        [7]
+  contractType: NFTContractTypeName.default, // other contract variation (custom or forked) [8]
+  autoApprove: true,                         // purely music or mixed-media NFT collection? [9]
+  typeMetadata: {
+    overrides: {                             // special treatment of track metadata        [10]
+      // track: { ... }                      //
+      // artist: { ... }                     //
+      // type: { ... }                       //
+      // extractor: { ... }                  //
+    }
+  }
+}
+```

.github/ISSUE_TEMPLATE/new-platform-integration-evaluation.md

@@ -0,0 +1,22 @@
+---
+name: New platform integration evaluation
+about: Form for evaluating the integration of a new platform into the Spindexer
+title: "[NEW PLATFORM INTEGRATION]"
+labels: ''
+assignees: ''
+
+---
+
+### Motivation
+
+When a request for new platform integration into the Spindexer is made we need to technically evaluate the request upfront.
+
+A developer should complete the checklist in this template to evaluate the integration.
+
+### Checklist for evaluation of integration request
+
+- [ ]  Platform/Artist Name:
+- [ ]  Chain:
+- [ ]  Would this integration require building new general feature(s) in the Spindexer?
+- [ ]  Would this integration require new specialized feature(s) in the Spindexer?
+- [ ]  Which mapper / extractors are appropriate for both tracks and artists?

README.md

@@ -58,7 +58,7 @@ watch -n 60 "yarn start >> ./log" 2>&1
 Sometimes things fail (eg: an offchain API is down). This is fine and expected. Things should continue as expected on the next run of the script. Most NFTs/Tracks/Platforms that cause failure/errors are moved into an error queue and retried a few times after some delay so that they don't block progress for the rest of the indexer.
 
 When experimenting or testing new PRs that may have breaking changes, you may want to reset the db often or test rebuilding on specific tables.
-There are various commands that help with this:
+There are various common commands that help with this:
 
 ```
 // This will reset the database to the most recent backup. Useful for testing just the impact of your PR if it is merged and deployed as is.
@@ -71,6 +71,8 @@ yarn reset-db-minimal
 yarn clear-track-tables
 ```
 
+See [Common Operations](./docs/common_operations.md) for more examples, or simply check out what's available in [package.json](./package.json).
+
 ## Design Goals
 There are a few design goals for the system:
  - Be fast, up to date and have low latency liveness
@@ -84,8 +86,9 @@ There are a few design goals for the system:
 ## Contributing
 The repo is still early and not hyper-polished or perfectly documented. Contributor guidelines are not ready yet, clear development docs and style/standard expectations are not extremely well documented yet. Interfaces are not well documented yet. Keep this in mind, so if you'd like to contribute:
  - First, reach out on Discord and connect with the team (https://discord.gg/8YS3peb62f) so we can help guide you in contributing
- - Read through the [Architecture](./Architecture.md) and [Ingestion](./Ingestion.md) docs which have some more details on the architecture and concepts used in the repo
+ - Read through the [Architecture](./docs/architecture.md) and [Ingestion](./docs/ingestion.md) docs which have some more details on the architecture and concepts used in the repo
  - Try set things up yourself, test it out, read the code.
  - Read the code some more, especially getting familiar with the [Processor](./src/types/processor.ts), [Trigger](./src/types/trigger.ts) and [Platform](./src/types/platform.ts) types and how they're used as interfaces :)
  - Check out the Github Project and Github Issues, still being improved
  - A note on migrations: If your PR includes a schema migration with new or changed fields, make sure it calls the updateViews method to update the corresponding views that are exposed in the API
+ - If you'd like to help with adding an artist's custom contract to the Spindexer, check out the [Basic Integration Guide](./docs/basic_integration_guide.md) or reach out to us

docs/basic_integration_guide.md

@@ -0,0 +1,132 @@
+# Basic Integration Guide for Artist's Tracks/Albums/Collections
+
+This is a comprehensive guide for what we call a "simple integration".
+
+These are often simple requests from Web3 artsts to index their latest track, a full album, or some kind of collection of NFTs that isn't already being indexed by Spindexer.
+
+### Starting Off
+
+Assuming you have nothing more than a URL pointing to a Web3 artist, their tracks, their album, or an NFT collection, your goals are simple to start with:
+
+1. Explore and collect details about the artist integration
+2. Understand whether existing code can process the music NFTs
+3. Estimate the scope of work
+
+## Exploring and Collecting Details
+
+### Finding good sources of information
+First, visit the submitted artist URL. Poke around and get a sense of where you'll find your primary sources of technical information.
+
+Some examples include:
+
+- https://opensea.io/
+- https://www.magiceden.io/
+
+Once you find a solid primary resources, search for the contract on [etherscan.io](https://etherscan.io) or [solscan.io](https://solscan.io) and begin collecting information for building out a new integration.
+
+### Which details to collect?
+The following key information is needed to build a simple integration. It is based on common integrations found in `src/db/migrations` .
+
+*Hint: Use the annotation numbers in the example below to find out more about a particular detail.*
+
+```ts
+// Use a `platform` to represent the artist or artist collaboration.
+const EXAMPLE_PLATFORM: MusicPlatform = {
+  id: '0x719...',                              // artist/artist-collaboration address [1]
+  type: MusicPlatformType['x-track-y-prints'], // # of tracks & # of NFTs minted      [2]
+  name: 'danielallan.xyz',                     // artist's official website           [3]
+}
+
+// Use an `nftFactory` to represent the contract which houses the album/collection/tracks.
+const EXAMPLE_ALBUM_OR_TRACK_COLLECTION: NftFactory = {
+{
+  id: '0x141...',                            // album/collection/track address              [4]
+  startingBlock: '15151004',                 // block number before contract inception      [5]
+  platformId: EXAMPLE_PLATFORM.id,           // new/existing platform id                    [6]
+  standard: NFTStandard.ERC721,              // contract type (relates to the chain)        [7]
+  contractType: NFTContractTypeName.default, // other contract variation (custom or forked) [8]
+  autoApprove: true,                         // purely music or mixed-media NFT collection? [9]
+  typeMetadata: {
+    overrides: {                             // special treatment of track metadata        [10]
+      // track: { ... }                      //
+      // artist: { ... }                     //
+      // type: { ... }                       //
+      // extractor: { ... }                  //
+    }
+  }
+}
+```
+
+## Understanding Pre-Existing Processing Strategies
+
+After having collected and considered details about the artist integration, it is important to understand which of the details fit nicely into the existing platform processors in [2].
+
+Also, become familiar with the variety of custom platform processors in `src/types/platform-types.ts` to look for common patterns in dealing with any idiosyncracies.
+
+
+## Estimating Scope of Work
+
+This last part should be much easier to do with all the details and thought from the first few steps (exhausting as they were).
+
+Generally:
+- the more that a "simple integration" overlaps with the existing platform processors in [2], the smaller the scope of work
+- more custom and non-standard the contract, the more likely for the scope to grow
+- when metadata for the NFT is difficult to restructure, non-standard, or entirely custom to incorporate, the greater the scope of work
+
+It is also important to know what to expect after the integration is complete, so try jot down the number of tracks that are expected to be added to the Spindexer.
+
+---
+
+## Appendix: Exploring & Collecting Details Hints + FAQ
+
+[1] Artist/artist-collaboration Address
+
+- Firstly, check whether the Artist ('platform') already exists from a previous integration, and avoid creating a new one by simply referring to the previous 'platform'
+- Otherwise, find the address of an artist by looking into an NFT, finding details on the artist, and looking for a link to their Ethereum/Solana wallet address. This will be different from the NFT address and is often linked to a `.eth` ENS record.
+
+[2] Number of tracks & number of NFTs minted
+
+- Looking at the collection on OpenSea, take a look at the structure of the collection. Depending on whether there are repeated NFTs in the collection, you can try to categorize the collection to use some of our existing platform processors:
+  - Many different tracks and non-repeating images - `MusicPlatformType['multi-track-multiprint-contract']`
+  - Many tracks with many repeated images - `MusicPlatformType['multi-track-multiprint-contract']`
+  - Single track with many repeated images - `MusicPlatformType['single-track-multiprint-contract']`
+- If it doesn't seem to fit the categorization, a custom mapper might be needed
+
+[3] Artist's official website
+
+- The artist/collab's official website is usually shown on an NFT or the collection. Otherwise use a search engine to find the artist's official website or platform.
+
+[4] Album/collection/track address
+
+- For an album or a collection, this is usually found via an Etherscan/Solscan link on the page via OpenSea.
+
+[5] Block number before contract inception
+
+- Looking at the collection through Etherscan/Solscan, click through to the "at txn" for the "Contract Creator" to find the contract's starting
+block. Subtract 1 from this block number and you've got your block number before contract inception.
+
+[6] New/existing platform id
+
+- Relates to the platform described in [1].
+
+[7] Contract type (relates to the chain)
+
+- Check the "Contract" tab on Etherscan/Solscan. You should see what kind of contract has been deployed. Generally, Ethereum contracts will follow ERC721 contracts or some similar custom implementation (see [8]) while Solana contracts follow Metaplex.
+- We currently support ERC721 and Metaplex NFT standards. ERC1155 contracts coming soon!
+
+[8] Other contract variation (custom or forked)
+
+- Standard ERC721 contracts are usually deployed for music NFTs, but some platforms like Nina and Zora have custom implementations. Check the "Contract" tab on Etherscan/Solscan to skim read the contract's code.
+
+[9] Purely music or mixed-media NFT collection?
+
+- Because we're mostly interested in indexing Web3 music and audio (currently), we'd like to avoiding indexing contracts that are purely for images or other non-audio media.
+- Along similar lines, some NFT collections contain snippets of audio (voice notes, goats bleeting, etc) that we likely don't want to include in our music library.
+- If either of these seem to be the case in the collection, a good strategy is to switch `autoApprove` off and to manually approve the desired NFTs using our [approvals mechanism](https://github.com/spinamp/spindexer/pull/78).
+
+[10] Special treatment of track metadata
+
+- To correctly index Web3 music NFTs, we also need to know details about the audio track, including its name, description, source URL, etc. These metadata fields are not standardized however, so inspecting the NFT metadata is critical.
+- An example of NFT metadata can be found on a particular NFT in OpenSea, in its "Details" section usually in a link labelled "TokenId".
+- Inspect the data structure to try to cross-reference it to the way a `ProcessedTrack` is compiled within `MapTrack` for the generalized strategies in section [2].
+- If the structure doesn't fit exactly to the existing mappers, consider implementing and extending the library of field extractors in `fieldExtractor.ts` and using them in the `overrides: { extractors: { ... } }` type metadata.