Remove residual fragments from failed video

[deldesir]

When a video failed due to unavailable fragments, a subsequent redownload will try to use the residual fragments from the previous download. This is because the download process did not go through to the point to have a video file and a thumbnail downloaded. Since the media.path did not synchronize with the actual calibre-web book path and no timestamp wasn't applied to the media.webpath, these residual fragments affects subsequent redownloads, thus their removal.

[holta]

Awesome if diagnostic understanding of yt-dlp and xklb is progressing!

(Any prior tickets we should link to for community context?)

[holta]

@EMG70 just FYI progress is cooking, as we try to release for you ASAP:

1. This PR (Remove residual fragments from failed video #157) is also high priority to be tested and merged, alongside:

   • PR Enable subsequent [re]downloads of playlists #151
   • PR Remove residual [duplicate] YouTube Shorts urls from database #155 might still need pytest testing? (Jacob Chapman requested that.)

2. These 2 URLs from Subsequent playlists redownloads fail #150 appear to be most useful for testing playlist redownloading:

   • https://www.youtube.com/playlist?list=PLGjplNEQ1it_oTvuLRNqXfz_v_0pq6unW 14 videos
   • https://www.youtube.com/playlist?list=PLfEr2kn3s-bo4LwlbyZugHPavhcdW8YMC 97 videos

   Whereas these 2 other URLs might not reproduce the problem??

   • https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist ✅
   • https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl - 53 short videos in playlist ✅

ASIDE: @deldesir tested different versions of yt-dlp a month ago, and they generally ALL WORKED — so let's not over-interpret this 1 data point on #158:

• Playlists did not fully download on newly built IIAB or upgraded server — 'NoneType' object cannot be interpreted as an integer — failed to download: Download appears to be stuck. #158 (comment) (Apr 30)
• https://github.com/deldesir/tests/blob/main/test_yt_dlp_tbr.py (May 1)

[deldesir]

I tried to downloaded https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi (102 videos) into t4 (Ubuntu 24.10) using #151, #155, #157. This 9-hour video failed to download:

Download: https://www.youtube.com/watch?v=9M4XKi25I2M failed to download: 'NoneType' object is not subscriptable

[holta]

Full results (from VM t4 above) — these 7 of 100 videos failed to download:

• 8:54:03 https://www.youtube.com/watch?v=9M4XKi25I2M
• 9:57 https://www.youtube.com/watch?v=dIu_C5Akino
• 6:56 https://www.youtube.com/watch?v=QeXmEeHuZ7w
• 8:12 https://www.youtube.com/watch?v=tU4JB2WvBVY
• 4:20 https://www.youtube.com/watch?v=AeS5OSKdaRM
• 7:09 https://www.youtube.com/watch?v=tzHecQr-LUc (datasette /library/calibre-web/xklb-metadata.db -h 0.0.0.0 -p 8080 shows that this is the ONLY 1 OF 7 where the video itself truly failed to download)
• 6:01 https://www.youtube.com/watch?v=Z5yuTVP2y_w

The other 6 videos were not successfully copied in the filesystem — and likewise each has the wrong path in xklb-metadata.db (/library/downloads/calibre-web instead of /library/calibre-web). These 6 failures might be a timeout issue, possibly related to... ?

• Handle hanging due to unavailable fragments #137

CLARIF / JUST FYI: Only 100 of 102 were attempted, due to

calibre-web/cps/constants.py

Lines 51 to 52 in 63ff431

	# Maximum number of videos to download, from a playlist or channel
	MAX_VIDEOS_PER_DOWNLOAD = 100

[deldesir]

[holta]

@deldesir and I agreed to dig deeper before merging this PR:

We need to try to understand the above timeout-like issue, which intermittently prevents videos from being copied from /library/downloads/calibre-web to /library/calibre-web (and likewise fails to update their path in xklb-metadata.db).

Very likely related:

• Handle hanging due to unavailable fragments #137
• Some videos failed to move [from /library/downloads/calibre-web] to /library/calibre-web #165

[holta]

@deldesir resolve this merge conflict?

(To keep this PR #157 discussion active!)

[holta]

Somewhat related PR:

• PR Adjust error message for stuck/slow videos [if elapsed_time >= timeout, e.g. due to unavailable fragments &/or xklb media_check] #223