Query captions/subtitles [e.g. video transcripts] from xklb-metadata.db

[deldesir]

🚀 Pull Request Overview:
This is an alternative PR to #152 designed for A/B testing. It allows searching querying xklb-metadata.db. The video titles ids returned as results are used to enhance Calibre-Web's simple search.

cf. https://github.com/chapmanjacobd/library/blob/main/xklb/mediadb/search.py

📋 Checklist:

• Tested the changes on Ubuntu 24.04 with https://www.youtube.com/watch?v=V8Bn0UPpGCw using xklb v2.9.036

root@box:~# lb search /library/calibre-web/xklb-metadata.db phytoplankton --to-json
8 captions
Is this the most valuable thing in the ocean? - David Biello - /library/calibre-web/TED-Ed/Is this the most valuable thing in the ocean_ - David Biello (9)/Is this the most valuable thing in the oce - TED-Ed.mp4
    1:21 we need to look at some of the ocean’s most ubiquitous organisms:
         phytoplankton.
    1:41 phytoplankton are everywhere.. and phytoplankton blooms are large
         enough to be seen from space.. These phytoplankton then become
         food for countless microscopic grazers,
    2:11 But when phytoplankton die, their bodies can sink far below the
         surface,
    2:27 whales consume huge amounts of these phytoplankton predators.
    3:16 Phytoplankton’s massive photosynthetic blooms. phytoplankton
         helps sequester huge amounts of carbon—. According to one report,
         phytoplankton are estimated to capture four times. every 1%
         increase in phytoplankton population
    4:02 researchers are working to increase phytoplankton populations
    4:17 than supporting the ocean’s natural phytoplankton farmers.

📌 Testing scenario:

• Download a single video or playlist
• Use the search field to search for some word or phrase present in the subtitles. You might need to watch a video to get some terms ;-)

Related to #152

cc @EMG70

[holta]

Related:

• PR Support searching videos [subtitles, captions, transcripts?] #152
• Search videos by subtitles/captions/transcripts [AND video title AND video description AND video author AND video publisher?] #197

[holta]

Should this PR be tested more broadly while comparing it to PR #152 now, as suggested at... ?

• Support searching videos [subtitles, captions, transcripts?] #152 (comment)

(Or is this PR still a draft for good reason?)

[deldesir]

It's still a draft. Trying to emulate xklb's internals here but my tests aren't conclusive yet due to a misunderstanding of how fts works. I am refactoring this completely to mirror (steal) xklb's code, then will remove the features we don't want for simplicity sake. Worth it as I am learning a lot by doing this.

[deldesir]

Ready for testing.

[holta]

Ready for testing.

Thanks!

Can you summarize the differences from PR #152 ?

This is an alternative PR to #152 designed for A/B testing. It allows searching querying xklb-metadata.db

Does this PR #244 search both metadata.db and xklb-metadata.db for completeness?

[deldesir]

Ready for testing.

Thanks!

Can you summarize the differences from PR #152 ?

This PR doesn't run any command in the background to get its results output to STDOUT, #152 runs a command.

This is an alternative PR to #152 designed for A/B testing. It allows searching querying xklb-metadata.db

Does this PR #244 search both metadata.db and xklb-metadata.db for completeness?

No, as explained in #197 , this PR only searches xklb-metadata.db. Completeness is assured by calibre-web internal search which merges its own results with this PR's ones.

[holta]

Can you summarize the differences from PR #152 ?

This PR doesn't run any command in the background to get its results output to STDOUT, #152 runs a command.

@deldesir explain in a bit more detail please, for everyone?

How/why is this approach (PR #244) likely preferred over PR #152 ?

[holta]

@EMG70 would you have time to test this PR?

(And compare it to your impressions of PR #152 !)

[EMG70]

I will test later on Today.

[deldesir]

This PR code does not execute any subprocess calls or run any external commands, #152 does. This PR is focused on querying and processing data within xklb-metadata.db, manipulating dictionaries, and formatting time strings.

The code is designed to interact with captions and media information. It return titles of media where captions match a search term.

This PR's approach is preferred because all operations are handled in-memory within Python without invoking any subprocesses (#152 invokes a subprocess to run lb search <db> <term> --to-json)

[deldesir]

Thanks again @EMG70. It looks like PR #259 is not pulled properly. Can you double check with the command:
git pull https://github.com/deldesir/calibre-web deldesir-xklb-schema --no-rebase --no-edit

Also run systemctl restart calibre-web after.

[EMG70]

After running PR#259
root@box:/usr/local/calibre-web-py3# git pull https://github.com/deldesir/calibre-web deldesir-xklb-schema --no-rebase --no-edit
error: Pulling is not possible because you have unmerged files.
hint: Fix them up in the work tree, and then use 'git add/rm '
hint: as appropriate to mark resolution and make a commit.
fatal: Exiting because of an unresolved conflict.
root@box:/usr/local/calibre-web-py3#

[deldesir]

Can you do git status so I can know which file contains conflicts? Feel free to post the output or screenshot here.

[EMG70]

root@box:/usr/local/calibre-web-py3# git status
On branch master
Your branch is ahead of 'origin/master' by 44 commits.
(use "git push" to publish your local commits)

You have unmerged paths.
(fix conflicts and run "git commit")
(use "git merge --abort" to abort the merge)

Unmerged paths:
(use "git add ..." to mark resolution)
both modified: cps/tasks/download.py

Untracked files:
(use "git add ..." to include in what will be committed)
bin/
include/
lib/
lib64
pyvenv.cfg

no changes added to commit (use "git add" and/or "git commit -a")
root@box:/usr/local/calibre-web-py3#

[deldesir]

Thanks. I would like to see one more output or screenshot to pin the problem.

Open cps/tasks/download.py by running nano /usr/local/calibre-web-py3/cps/tasks/download.py.
You'll see sections marked with conflict markers.

Copy paste these sections. They look like:

<<<<<<< HEAD 
======= 
>>>>>>>

A screeshot is ok if copy pasting is not possible

[EMG70]

 <<<<<<< HEAD
 57
 58                 # Database operations
 59                 with sqlite3.connect(XKLB_DB_FILE) as conn:
 60                     try:
 61                         # requested_file = conn.execute("SELECT path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()[0]
 62                         self.media_id, requested_file = conn.execute("SELECT id, path FROM media WHERE webpath = ? AND path NOT LIKE 'http%'", (self.media_url,)).fetchone()
 63
 64                         # Abort if there is not a path
 65                         if not requested_file:
 66                             log.info("No path found in the database")
 67                             error = conn.execute("SELECT error, webpath FROM media WHERE error IS NOT NULL").fetchone()
 68                             if error:
 69                                 log.error("[xklb] An error occurred while trying to download %s: %s", error[1], error[0])
 70                                 self.message = f"{error[1]} failed to download: {error[0]}"
 71                             else:
 72                                 log.error("%s failed to download: No path or error found in the database (likely the video failed due to unavailable fragments?)", self.media_url)
 73                                 self.message = f"{self.media_url_link} failed to download: No path or error found in the database (likely the video failed due to unavailable fragm>
 74                                 self.media_id = conn.execute("SELECT id FROM media WHERE webpath = ?", (self.media_url,)).fetchone()[0]
 75                                 conn.execute("DELETE FROM media WHERE webpath = ?", (self.media_url,))
 76                                 conn.execute("DELETE FROM captions WHERE media_id = ?", (self.media_id,))
 77                             return
 78                     except sqlite3.Error as db_error:
 79                         log.error("An error occurred while trying to connect to the database: %s", db_error)
 80                         self.message = f"{self.media_url_link} failed to download: {db_error}"
 81
 82                     self.message = self.message + "\n" + f"Almost done..."
 83                     response = requests.get(self.original_url, params={"requested_file": requested_file, "current_user_name": self.current_user_name, "shelf_id": self.shelf_id, "m>
 84                     if response.status_code == 200:
 85                         log.info("Successfully sent the requested file to %s", self.original_url)
 86                         file_downloaded = response.json()["file_downloaded"]
 87                         self.message = f"Successfully downloaded {self.media_url_link} to <br><br>{file_downloaded}"
 88                         new_video_path = response.json()["new_book_path"]
 89                         new_video_path = next((os.path.join(new_video_path, file) for file in os.listdir(new_video_path) if file.endswith((".webm", ".mp4"))), None)
 90                         # 2024-02-17: Dedup Design Evolving... https://github.com/iiab/calibre-web/pull/125
 91                         conn.execute("UPDATE media SET path = ? WHERE webpath = ?", (new_video_path, self.media_url))
 92                         conn.execute("UPDATE media SET webpath = ? WHERE path = ?", (f"{self.media_url}&timestamp={int(datetime.now().timestamp())}", new_video_path))
 93                         self.progress = 1.0
 94                     else:
 95                         log.error("Failed to send the requested file to %s", self.original_url)
 96                         self.message = f"{self.media_url_link} failed to download: {response.status_code} {response.reason}"
 97
 98                 conn.close()
 99
100 =======
101                 self._post_process_download()
102 >>>>>>> 95a0d554e50fef881f129b4ed2d929249dbffd17

[deldesir]

I think I understand what happened. I'll be back with a solution in a couple of minutes. Thanks for your patience.

[deldesir]

It shouldn't be you to do this, but I'll ask you to fix your install with the instructions below. I must tell you the problem was caused by PR #255 proposing unecessary changes for download.py, not because you did something wrong. I understand now the order matters when testing PRs so I did my part by adjusting PR #255 to prevent this issue for others.

• Open the download.py if not already opened with nano /usr/local/calibre-web-py3/cps/tasks/download.py
• Remove <<<<<<< HEAD (should be at line 56)
• Remove lines 100, 101, 102
• Save and Exit from nano editor:
  • press Ctrl + X.
  • you will be prompted to save, press Y to confirm.
  • Press Enter to accept the current file name.
• Stage the file by doing git add cps/tasks/download.py
• Commit your changes by running git commit -m "Resolved conflict in download.py"
• Pull PR Begin to establish 1:1 mapping between media_id (xklb-metadata.db) <-> Calibre-Web's book_id (metadata.db) #255 again by running git pull https://github.com/deldesir/calibre-web deldesir-patch-23 --no-rebase --no-edit
• Pull PR Model xklb-metadata.db using SQLAlchemy #259 again by running git pull https://github.com/deldesir/calibre-web deldesir-xklb-schema --no-rebase --no-edit
• Restart Calibre-Web by running systemctl restart calibre-web

[holta]

It shouldn't be you to do this

Indeed! This is a struggle we're currently sorting out, and let's all be patient getting this right! And in future:

• Testing volunteers should NEVER get involved with merge conflicts. ✅
• Test Plan / Test Instructions need to be FULLY SMOKE-TESTED in advance. ✅

[EMG70]

I have followed your instructions .
SUDO IIAB-DIAGNOSTICS - https://paste.centos.org/view/ba1d5609
calibre-web still not reachable.

root@box:/usr/local/calibre-web-py3# systemctl restart calibre-web
root@box:/usr/local/calibre-web-py3# systemctl status calibre-web
× calibre-web.service - Calibre-Web
Loaded: loaded (/etc/systemd/system/calibre-web.service; enabled; preset: enabled)
Active: failed (Result: exit-code) since Sat 2024-10-19 22:07:10 BST; 6s ago
Duration: 1.330s
Process: 2247 ExecStart=/usr/local/calibre-web-py3/bin/python3 /usr/local/calibre-web-py3/cps.py -p /library/calibre-web/config/app.db (code=exited, status=1/FAILURE)
Main PID: 2247 (code=exited, status=1/FAILURE)
CPU: 1.328s

Oct 19 22:07:10 box python3[2247]: File "/usr/local/calibre-web-py3/cps/init.py", line 42, in
Oct 19 22:07:10 box python3[2247]: from . import ub, db
Oct 19 22:07:10 box python3[2247]: File "/usr/local/calibre-web-py3/cps/db.py", line 52, in
Oct 19 22:07:10 box python3[2247]: from .services.xb_utils import CaptionSearcher
Oct 19 22:07:10 box python3[2247]: File "/usr/local/calibre-web-py3/cps/services/xb_utils.py", line 9, in
Oct 19 22:07:10 box python3[2247]: from cps.glue_db import GlueDB, MediaBooksMapping
Oct 19 22:07:10 box python3[2247]: ModuleNotFoundError: No module named 'cps.glue_db'
Oct 19 22:07:10 box systemd[1]: calibre-web.service: Main process exited, code=exited, status=1/FAILURE
Oct 19 22:07:10 box systemd[1]: calibre-web.service: Failed with result 'exit-code'.
Oct 19 22:07:10 box systemd[1]: calibre-web.service: Consumed 1.328s CPU time.
root@box:/usr/local/calibre-web-py3#

[deldesir]

Ok. Let's call it a failed test for now. On my side, I will be following the instructions step by step to ensure we eat the same food👌. I'll provide some feedback once done to better assist you. Thanks for your hard testing and patience.

[holta]

following the instructions step by step to ensure we eat the same food👌

Perfect. Thank you.

[deldesir]

I did the test following the instructions step by step: https://paste.centos.org/view/dc30dfdf
@EMG70, please do the test again when possible.

You can either do it with a new VM or do it with the current one instead by doing

• cd /usr/local/
• mv calibre-web-py3 calibre-web-py3.OLD
• cd /opt/iiab/iiab
• ./runrole --reinstall calibre-web

... and then try the pulls again:

cd /usr/local/calibre-web-py3
git pull https://github.com/deldesir/calibre-web deldesir-search-captions --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-22 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-23 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-24 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-25 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-27 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-xklb-schema --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-31 --no-rebase --no-edit
iiab-update -f

Above instructions worked for me, but it they fail for you, feel free to let me know:

[EMG70]

Just a quick note.I have installed another fresh VM ,this time a 24.10.It worked OK as per above original instuctions.I know @holta had suggested not to use 24.10.
Could this issue be confined to 24.10 LTS?
SUDO IIB-DIAGNOSTICS https://paste.centos.org/view/bc3e5380

Download of a single video failed though.

[EMG70]

I did the test following the instructions step by step: https://paste.centos.org/view/dc30dfdf @EMG70, please do the test again when possible.

You can either do it with a new VM or do it with the current one instead by doing

* ` cd /usr/local/`

* `mv calibre-web-py3 calibre-web-py3.OLD`

* `cd /opt/iiab/iiab`

* `./runrole --reinstall calibre-web`

... and then try the pulls again:

cd /usr/local/calibre-web-py3
git pull https://github.com/deldesir/calibre-web deldesir-search-captions --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-22 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-23 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-24 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-25 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-27 --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-xklb-schema --no-rebase --no-edit
git pull https://github.com/deldesir/calibre-web deldesir-patch-31 --no-rebase --no-edit
iiab-update -f

Above instructions worked for me, but it they fail for you, feel free to let me know:

Am on it now.

[deldesir]

Just a quick note.I have installed another fresh VM ,this time a 24.10.It worked OK as per above original instuctions.I know @holta had suggested not to use 24.10. Could this issue be confined to 24.10 LTS? SUDO IIB-DIAGNOSTICS https://paste.centos.org/view/bc3e5380

Download of a single video failed though.

Glad the instructions worked for 24.10.
Yes, you're right for the download failure as per issue #267

[holta]

• WARNING: Avoid Ubuntu 24.10 for now, as all video downloads appear to fail due to "Download to IIAB" button fails every time on Ubuntu 24.10 — "An error occurred during the subprocess execution: 'NoneType' object is not subscriptable" and later "failed to download: None" — perhaps b/c xklb marked your video as deleted because it failed to download in previous tries? #267 !

▶️ @EMG70 Please AVOID Ubuntu 24.10 as explained above (yesterday): #244 (comment)

▶️ Testing needs to begin on Ubuntu 24.04 "so we're all eating the same food."

[EMG70]

Update after @deldesir's latest instructions:
sudo iiab-diagnostics - https://paste.centos.org/view/4f127ac4

calibre-web now working OK.

step 3. root@box:/usr/local/calibre-web-py3# ls /library/calibre-web/xklb-metadata.db
/library/calibre-web/xklb-metadata.db
root@box:/usr/local/calibre-web-py3#

Step 4. ✔️
step 5. ✔️

Step 6. Also verify if metadata are extended and the media.path is actually a path instead of a URL for the required video — meaning that the download.py refactoring was effective, and functionality is preserved.

Step 7.To further test, play a video and note the number in the URL for the video played from within Calibre-Web (e.g. for http://<IIAB_HOSTNAME_OR_IP>/books/read/31.mp4, the video ID is 31). Note also the title of the video, and note the media ID associated with it in iiab-glue.db.

Step 8.Find 2 videos with overlapping words in the title, but different captions

Step 9.Search for a term that's in the captions of one video but not the other. Only the video where the caption matches should return.

NB: I had downloaded more than 2 videos so results may not be as clear as expected,hope that gives an idea.

[EMG70]

• WARNING: Avoid Ubuntu 24.10 for now, as all video downloads appear to fail due to "Download to IIAB" button fails every time on Ubuntu 24.10 — "An error occurred during the subprocess execution: 'NoneType' object is not subscriptable" and later "failed to download: None" — perhaps b/c xklb marked your video as deleted because it failed to download in previous tries? #267 !

▶️ @EMG70 Please AVOID Ubuntu 24.10 as explained above (yesterday): #244 (comment)

▶️ Testing needs to begin on Ubuntu 24.04 "so we're all eating the same food."

Noted.

[deldesir]

Instructions updated to include running PIPX_BIN_DIR=/usr/local/bin pipx install datasette in Step 5.

Here are the complete instructions:

How to test

1. Install a brand new IIAB with Calibre-Web. We'd welcome tests on Ubuntu 24.04, Raspberry Pi OS or any other Debian-based distro running on old laptops.

   • WARNING: Avoid Ubuntu 24.10 for now, as all video downloads appear to fail due to "Download to IIAB" button fails every time on Ubuntu 24.10 and 25.04 pre-releases — originally "An error occurred during the subprocess execution: 'NoneType' object is not subscriptable" — and now "failed to download: None" #267 !

2. Pull in the 8 PRs: (background: How do I test a pull request (PR) ?)

   sudo -i
   cd /usr/local/calibre-web-py3
   git pull https://github.com/deldesir/calibre-web deldesir-search-captions --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-patch-22 --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-patch-23 --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-patch-24 --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-patch-25 --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-patch-27 --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-xklb-schema --no-rebase --no-edit
   git pull https://github.com/deldesir/calibre-web deldesir-patch-31 --no-rebase --no-edit
   iiab-update -f
   

   Just FYI this means you should be testing these 8 PRs, all together:

   • Query captions/subtitles [e.g. video transcripts] from xklb-metadata.db #244
   • Refactor video_metadata to use xb #254
   • Begin to establish 1:1 mapping between media_id (xklb-metadata.db) <-> Calibre-Web's book_id (metadata.db) #255
   • Create xb_utils.py [to query/manipulate xklb-metadata.db (schema in xb.py) + iiab-glue.db (schema in glue-db.py)] #256
   • Refactor metadata_extract.py #257
   • Refactor download.py #258
   • Model xklb-metadata.db using SQLAlchemy #259
   • Modeling iiab-glue.db to enable mapping of books.id (metadata.db) and media.id (xklb-metadata.db) #263

3. Before downloading a video, do ls /library/calibre-web/xklb-metadata.db in your Terminal to check for the db existence. If it exists, that means the SQLAlchemy successfully created it, attesting to the effectiveness of the database modeling.

4. Submit a YouTube video URL via the “Download to IIAB” button and monitor the “Tasks” page to check whether metadata fetching is reported.

5. On success, install datasette by running: PIPX_BIN_DIR=/usr/local/bin pipx install datasette

   • Then review the extraction with datasette /library/calibre-web/xklb-metadata.db -h 0.0.0.0 -p 8080 by browsing to http://<IIAB_HOSTNAME_OR_IP>:8080
   • If metadata are present for the required video, that means the metadata_extract.py refactoring was effective and functionality is preserved.
   • Remember to type CTRL-c in the Linux Terminal where datasette is running, to end the datasette program as soon as you no longer need it, to free up port 8080 !

6. Also verify if metadata are extended and the media.path is actually a path instead of a URL for the required video — meaning that the download.py refactoring was effective, and functionality is preserved.

7. To further test, play a video and note the number in the URL for the video played from within Calibre-Web (e.g. for http://<IIAB_HOSTNAME_OR_IP>/books/read/31.mp4, the video ID is 31). Note also the title of the video, and note the media ID associated with it in iiab-glue.db.

   • In another terminal window, run datasette /library/calibre-web/iiab-glue.db -h 0.0.0.0 -p 8080
   • Open a new browser tab to go to http://<IIAB_HOSTNAME_OR_IP>:8080
   • Remember to type CTRL-c in the Linux Terminal where datasette is running, to end the datasette program as soon as you no longer need it, to free up port 8080 !
   • This media ID can be verified by inspecting xklb-metadata.db for its correctness. To do so with datasette, run datasette /library/calibre-web/xklb-metadata.db -h 0.0.0.0 -p 8080. Look for the ID row in media table and compare the title from the actual video with the record under media.title.

8. Find 2 videos with overlapping words in the title, but different captions.

9. Search for a term that's in the captions of one video but not the other. Only the video where the caption matches should return.

10. Wait 20mn, then download a video to check whether the erro "a transaction is already begun" is gone.

11. Do a search while a video is being downloaded to ensure that session management is working as expected.

[EMG70]

I will continue steps 6-9 tomorrow.

[holta]

I just fixed the instructions for installing datasette in the Test Plan just above:

• pipx install datasette ❌
• PIPX_BIN_DIR=/usr/local/bin pipx install datasette ✅
• PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install datasette if you want to follow our official instructions !

[deldesir]

@EMG70, did you get a 404 not found error when testing on 24.04. On my side, when I do the tests on Raspberry PI OS, I could not access Calibre-Web normally. I had to specify the Port 8083 to access it.

[deldesir]

Calibre-Web is now accessible on UNITTEST install on Raspberry Pi OS. All I had to do is:

cd /opt/iiab/iiab
./runrole network 

[EMG70]

@EMG70, did you get a 404 not found error when testing on 24.04. On my side, when I do the tests on Raspberry PI OS, I could not access Calibre-Web normally. I had to specify the Port 8083 to access it.

I have not yet tried latest PR tests on Raspberry PI.

[holta]

cd /opt/iiab/iiab
./runrole network

The official/recommended way of doing the above is to run the following as root: (the cd command is not necessary!)

iiab-network

(But possibly these 2 variables should be set in /etc/iiab/local_vars.yml first?)

network_install: True
network_enabled: True

[deldesir]

@EMG70 Test plan updated to check the fix for #271