SLC 1 (#87)

* test: add failing test for multiple uploads

* feature: keep track of multiple artists in uploads

* tweak: simplify money rounding for now

* test: add failing tests and test csv for bandcamp upload

* refactor: share csv header functionality

* feature: implement bandcamp upload and detection

* build: ignore data directory

* feature: clean datasets on import

* feature: clean null bytes and hidden tracking characters

* test: test that bandcamp files are processed correctly

* test: a few tests to cover the artist usage aggregation

* wip: restructure page using uploaded-files-list tag

* wip: style new structure

* wip: sort data

* wip: new class layout for artist usages

* feature: feature complete #51

* test: test for multiple artists

* Change titles and styling on homepage

* Basic UI styling v1

* Styling for multiple artists

* test: rename HTML fields

* Create homepage.md

* Update homepage.md

Headine and Subheading added.

* Update homepage.md

Additional content writing

* design: style tidy up

* wip: tidy up upload/statement usage

* test: refactor phpunit v10

* test: simplify tests for new codebase

* test: full ci tooling

* ci: test class dir, not src dir

* ci: php 8,2

* ci: wrap php version in quotes?

* ci: bump pdepend

* ci: use phpmd 2.13.0

* ci: use vendored version

* ci: use vendored version

* ci: add deployment back in

* ci: add deployment back in

* content: add missing assets

* tweak: client side files

* build: bump deps

* wip: slc1

* wip: api config

* test: upload classes including cargo

* test: improve phpmd and phpcs tests

* test: update dummy test data

* build: update dependencies

* wip: cargo product matching

* wip: optimise file upload fetch request for large files

* feature: lazy load images from spotify

* build: apply fix for domtemplate

* wip: temporarily disable ci deployment

* wip: search spotify by artist

* fix: filter product list by user
closes #93

* wip: rejig header and file-uploader component

* wip: fix spotify search

* feature: exit early if there's no matching artwork on spotify

* tweak: format search query

* tweak: update spotify api usage

* tweak: update spotify api usage (album match)

* tweak: improve upload process

* styling

* new styling

* loading page with animation and sheen

* wip: in memory database insertion

* feature: tidy with FilterQuery

* feature: set filtered after completion

* build: upgrade dependencies

* feature: cron script for background processing

* feature: loop within php script

* feature: move processing back in-line

* wip: get rid of debug code

* wip: fix sqlite syntax

* wip: lazy load - ignore case

* build: update dependencies

* fix font file naming convention

* design: geomanist font

* design: privacy policy page container

* wip: html in markdown

* design: markdown styling tweaks

* design: homepage

* design: footer zap width

* feature: TunecoreUpload

* design: homepage copy and images for SLC1 launch

* editorial: added contact details to homepage

* tweak: delete from database as well as clear file on disk

* fix: remove limit

* feature: bring back delete-me section

* feature: file clear/expire/extend

* feature: auto purge

* fix: cargo earning

* design: font size and 'delete-me' formatting

* design: uploads

* tweak: style change for old chrome

---------

Co-authored-by: Richard Birkin <richardbirkin@iBiff-Pro.lan>
Co-authored-by: Richard Birkin <richardbirkin@iBiff-Pro.local>
Co-authored-by: richardbirkin <115098450+richardbirkin@users.noreply.github.com>
Co-authored-by: rjbirkin <richard@rjbirkin.co.uk>

Author: 4 people

.github/workflows/ci.yml

@@ -5,6 +5,9 @@ on: [push]
 jobs:
   composer:
     runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        php: [ 8.2 ]
 
     steps:
       - uses: actions/checkout@v3
@@ -18,7 +21,7 @@ jobs:
       - name: Composer install
         uses: php-actions/composer@v6
         with:
-          php_version: '8.1'
+          php_version: ${{ matrix.php }}
 
       - name: Archive build
         run: mkdir /tmp/github-actions/ && tar -cvf /tmp/github-actions/build.tar ./
@@ -31,7 +34,13 @@ jobs:
 
   phpunit:
     runs-on: ubuntu-latest
-    needs: [composer]
+    needs: [ composer ]
+    strategy:
+      matrix:
+        php: [ 8.2 ]
+
+    outputs:
+      coverage: ${{ steps.store-coverage.outputs.coverage_text }}
 
     steps:
       - uses: actions/download-artifact@v3
@@ -44,15 +53,44 @@ jobs:
 
       - name: PHP Unit tests
         uses: php-actions/phpunit@v3
+        env:
+          XDEBUG_MODE: cover
         with:
-          php_version: '8.1'
+          php_version: ${{ matrix.php }}
           php_extensions: xdebug
-          configuration: test/phpunit/phpunit.xml
-          bootstrap: vendor/autoload.php
+          coverage_text: _coverage/coverage.txt
+          coverage_clover: _coverage/clover.xml
+
+      - name: Store coverage data
+        uses: actions/upload-artifact@v3
+        with:
+          name: code-coverage
+          path: _coverage
+
+  coverage:
+    runs-on: ubuntu-latest
+    needs: [ phpunit ]
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - uses: actions/download-artifact@v3
+        with:
+          name: code-coverage
+          path: _coverage
+
+      - name: Output coverage
+        run: cat "_coverage/coverage.txt"
+
+      - name: Upload to Codecov
+        uses: codecov/codecov-action@v3
 
   phpstan:
     runs-on: ubuntu-latest
-    needs: [composer]
+    needs: [ composer ]
+    strategy:
+      matrix:
+        php: [ 8.2 ]
 
     steps:
       - uses: actions/download-artifact@v3
@@ -66,12 +104,15 @@ jobs:
       - name: PHP Static Analysis
         uses: php-actions/phpstan@v3
         with:
+          php_version: ${{ matrix.php }}
           path: class/
-          level: 6
 
-  behat:
+  phpmd:
     runs-on: ubuntu-latest
-    needs: [composer]
+    needs: [ composer ]
+    strategy:
+      matrix:
+        php: [ 8.2 ]
 
     steps:
       - uses: actions/download-artifact@v3
@@ -82,17 +123,21 @@ jobs:
       - name: Extract build archive
         run: tar -xvf /tmp/github-actions/build.tar ./
 
-      - name: Local development server
-        run: mkdir www && php -S 0.0.0.0:8080 -t www vendor/phpgt/webengine/go.php &
-
-      - name: Behat tests
-        uses: php-actions/behat@v1
+      - name: PHP Mess Detector
+        uses: php-actions/phpmd@v1
         with:
-          config: test/behat/behat.yml
+          php_version: ${{ matrix.php }}
+          vendored_phpmd_path: vendor/bin/phpmd
+          path: class/
+          output: text
+          ruleset: phpmd.xml
 
-  dev-deploy:
+  phpcs:
     runs-on: ubuntu-latest
-    needs: [composer, phpunit, phpstan, behat]
+    needs: [ composer ]
+    strategy:
+      matrix:
+        php: [ 8.2 ]
 
     steps:
       - uses: actions/download-artifact@v3
@@ -103,11 +148,46 @@ jobs:
       - name: Extract build archive
         run: tar -xvf /tmp/github-actions/build.tar ./
 
-      - uses: php-actions/deploy-ssh@master
+      - name: PHP Code Sniffer
+        uses: php-actions/phpcs@v1
         with:
-          hostname: deploy-endpoint.dev.trackshift.app
-          user: deploy
-          path: /var/www/trackshift
-          path_owner: www-data.
-          ssh_key: ${{ secrets.deploy_ssh_key }}
-          after_command: ./vendor/bin/gt build
+          php_version: ${{ matrix.php }}
+          path: class/
+          standard: phpcs.xml
+
+#  dev-deploy:
+#    runs-on: ubuntu-latest
+#    needs: [ composer, phpunit, phpstan, phpcs, phpmd ]
+#
+#    steps:
+#      - uses: actions/download-artifact@v3
+#        with:
+#          name: build-artifact
+#          path: /tmp/github-actions
+#
+#      - name: Extract build archive
+#        run: tar -xvf /tmp/github-actions/build.tar ./
+#
+#      - uses: php-actions/deploy-ssh@master
+#        with:
+#          hostname: deploy-endpoint.dev.trackshift.app
+#          user: deploy
+#          path: /var/www/trackshift
+#          path_owner: www-data.
+#          ssh_key: ${{ secrets.deploy_ssh_key }}
+#          after_command: ./vendor/bin/gt build
+
+  remove_old_artifacts:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Remove old artifacts for prior workflow runs on this repository
+        env:
+          GH_TOKEN: ${{ github.token }}
+        run: |
+          gh api "/repos/${{ github.repository }}/actions/artifacts?name=build-artifact" | jq ".artifacts[] | select(.name == \"build-artifact\") | .id" > artifact-id-list.txt
+          while read id
+          do
+            echo -n "Deleting artifact ID $id ... "
+            gh api --method DELETE /repos/${{ github.repository }}/actions/artifacts/$id && echo "Done"
+          done <artifact-id-list.txt

.gitignore

@@ -1,2 +1,9 @@
 /vendor/
-.phpunit.result.cache
+/data/
+/ts.db
+/config.*.ini
+/www/
+.DS_Store
+access-token.dat
+asset/font/.DS_Store
+asset/.DS_Store

README.md

@@ -1,23 +1,39 @@
 A music royalties data aggregator
 =================================
 
-// TODO: Very short intro to the project and brief description of the codebase.
+We make releasing music easier.
 
-A tool that helps artists and record labels process and analyse the complex web of payment data from vendors.
+We are DIY artists, label people, software developers and music consumers who are building tools to give indies superpowers in the age of big data.
 
-Giving indies superpowers in the age of big data.
+"Giving indies superpowers in the age of big data."
+
+Dev todo
+--------
+
+Notes for Wednesday: the database speedup is immense. From 6 minutes to about 15 seconds.
+But there's still more to do, and ideally it should all be done within 1 second.
+So, here's how:
+
+- [x] When the file is uploaded, just store it in the Upload table.
+- [x] Introduce a new field, Upload.processedUsages
+- [x] In a background script, loop over all uploads that are not processed and extract their usages (then mark as processed)
+- [ ] Introduce another new field, Usage.processed
+- [ ] In another background script, loop over all usages that are unprocessed, finishing the job here.
+- [ ] The usage processor needs to match products and artists - rather than doing this individually in a loop, lookup the unique artist/product first, to cache the IDs (or create new ones), then it's possible to insert UsageOfProduct rows on bulk!
+- [ ] Then optimise further with a profiler. Ideally, a very large import should be completed before the page has chance to reload.
+- [ ] If a spinner is necessary, it should be put onto the three-checkbox page. It's also possible to know how many usages are left to process, so an ACTUAL progress bar is possible.
 
 Setup guide
 -----------
 
-// TODO: From scratch for Linux, Windows and Mac.
+TODO: From scratch for Linux, Windows and Mac.
 
 Running locally
 ---------------
 
-// TODO.
+TODO.
 
 Writing/running tests
 ---------------------
 
-// TODO.
+TODO.