Enable subsequent [re]downloads of playlists

[deldesir]

Fixes #150. This PR adds a timestamp to playlists' paths after they are downloaded.

[ 2024-05-30 SUMMARY: xklb generally does not support subsequent redownloading of playlists, so let's try this workaround! ]

[deldesir]

Tested with subsequent [re]downloading of a small playlist. Couldn't test with long playlists due to potential disk IO error.

[EMG70]

IIAB-DIAGNOSTICS - http://sprunge.us/SormHj?en

All single single videos and playlists failing to download.

https://www.youtube.com/watch?v=T3kDWpREiEI&list=PLoC8Q0moRTSjKkMZoVkAgHMJjkpUreleb&index=21e - single video
https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist
https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl - 53 short videos in playlist

[deldesir]

Hi @EMG70. Reading your log, it looks like xklb-metadata.db is missing. Can you check by doing ls /library/calibre-web/xklb-metadata.db using your terminal?

[holta]

Hi @EMG70. Reading your log, it looks like xklb-metadata.db is missing. Can you check by doing ls /library/calibre-web/xklb-metadata.db using your terminal?

Or run this as root...

apt install tree
cd /library/calibre-web
tree

[EMG70]

Hi @EMG70. Reading your log, it looks like xklb-metadata.db is missing. Can you check by doing ls /library/calibre-web/xklb-metadata.db using your terminal?
Here is the output

[deldesir]

Line 1090 in your log says you don't have XKLB installed. How come you have the database and no XKLB installed? Weird!

[deldesir]

To confirm run:
which lb

If it is found, do

lb --version

[EMG70]

Line 1090 in your log says you don't have XKLB installed. How come you have the database and no XKLB installed? Weird!

Allow me to reinstall and try again in case i missed something

[deldesir]

Please go ahead, no worry.

[EMG70]

To confirm run: which lb

If it is found, do

lb --version

sorry a bit late now,VM destroyed.

[EMG70]

Please check if all is ok before proceeding with testing.

[deldesir]

Looks good so far

[holta]

Plz post ideas(s) as to why the earlier XKLB install failed / disappeared, if possible?

[EMG70]

IIAB-DIAGNOSTICS - http://sprunge.us/tjeC5u?en

Downloading and video ranking showing OK .Subsequent playlist downloads all failed no matter number of videos in playlist,however a single video downloads well.

This playlist downloaded all 53 videos ok but failed on second attempt. https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl
https://www.youtube.com/playlist?list=PLiMWaCMwGJXlQj0BFEwew0fvhOPr833GS 7 videos fails

[deldesir]

I cannot reproduce your issue @EMG70, sorry. Both Playlists work for me, all videos were downloaded, except 1 (https://www.youtube.com/watch?v=7m4_kZOObzw). I am afraid you might be subject to some restrictions either by your ISP or YouTube itself

[EMG70]

You could be right.I have not done any testing since I changed ISP two weeks ago.
I will try and see what's going on.

[holta]

might be subject to some limitations either by your ISP or YouTube itself

We can and should arrange for @EMG70[*] to do the exact same test remotely (i.e. with a different ISP and different YouTube geographic restrictions) to confirm what exactly is happening and why.

[*] And others too, if possible!

@deldesir can you help spell out the exact steps of what most needs to be tested, to help everyone avoid ambiguity here?

[deldesir]

We're dealing with a download issue. We know we have a successful test if we provide 2-3 playlists one after the other and the following conditions are met for them:

• metadata are fetched and total duration is displayed
• videos are downloaded (a few failures are ok)
• shelves are created and populated

[holta]

We know we have a successful test if we provide 2-3 playlists one after the other and the following conditions are met [,,,]

Thanks for explaining.

Are we also trying to test: repeatedly entering the same playlist URL?

(i.e. clicking "Download to IIAB" each time, and entering the same YouTube playlist URL each time, a few times in succession)

[deldesir]

I looked into t5. Everything looks Ok. Can't spot the issue right now. I will delete xklb-metadata and run the playlists again.

[deldesir]

Thanks a lot for testing again. The UNIQUE constraint error is a database issue. I'll let you know more once I am done examining your log.

[EMG70]

Sorry did i post on wrong PR.I have just posted same on PR#150

[EMG70]

Thanks a lot for testing again. The UNIQUE constraint error is a database issue. I'll let you know more once I am done examining your log.

IIAB-DIAGNOSTICS - http://sprunge.us/Kv2wBX?en
https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist ✅
https://www.youtube.com/playlist?list=PLr6-GrHUlVf96NLj3PQq-tmEB6woZjwEl - 53 short videos in playlist ✅
97/102 Videos were downloaded OK.A few videos failed https://www.youtube.com/watch?v=4e3dIeP93E8 and https://www.youtube.com/watch?v=D2KvDBSZA0w although they do play well on Youtube.I am not sure why,see screenshots:

[deldesir]

For the 102 videos url (https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi - 102 short videos in playlist) I also had 3 failed videos:

• https://www.youtube.com/watch?v=dIu_C5Akino
• https://www.youtube.com/watch?v=EQCUdCG-eAA
• https://www.youtube.com/watch?v=4e3dIeP93E8

We have one in common: https://www.youtube.com/watch?v=4e3dIeP93E8

When I forced the redownload of any failed video I got the UNIQUE constraint failed: media.path error. It's because :
1- no error was reported by xklb for the failed video. It's like the video succeeded and even got a download/local "path"
2- since the video really failed, calibre-web did not have it, so no timestamp was added to the path

[holta]

Thanks @deldesir for explaining the progress:

Is targeted testing needed (e.g. from @EMG70) in coming days?

[deldesir]

Redownloading a playlist twice still doesn't work as expected. This will require some adjustments for sure. Hope I fix it today or tomorrow so Ed can battletest it.

[deldesir]

@EMG70 and @holta, I just got 100% success downloading and redownloading https://www.youtube.com/playlist?list=PL_c9BZzLwBRLVh9OdCBYFEql6esA6aRsi. What changed? No code additions, I only updated my branches based on iiab:master and resolved 2 merged conflicts that led to the closing of #146. I also notice the VM where failures from redownloading happened did not have this commit d9e4fc4. Examining the [ xklb-metadata.db ] confirmed this because the downloaded playlists do not have any timestamps added to them.

http://sprunge.us/lPRkPg

[holta]

@deldesir what do you suggest?

[deldesir]

I'd like a second look into this. I want to be sure this works for @EMG70 too.

[holta]

Thanks @deldesir:

Please make a recommendation if additional testing is required?

[deldesir]

A clean test environment @holta.
@EMG70, If you plan to test this again, ensure you don't include #152. I don't understand yet what went wrong so a clean VM with only #151 is recommended. I have to acknowledge we benefit from the enhancements made recently to yt-dlp and xklb so your test will probably use:

• yt-dlp 2024.04.09
• xklb 2.6.009

[EMG70]

I will test tomorrow and update you.

[deldesir]

@EMG70 and @holta, downloading channel with this PR is not solid yet due to "Shorts" not having time_uploaded. I'll let you know once I'm done handling this in a better way. For your tests, please consider playlists only for now.

[EMG70]

IIAB-DIAGNOSTICS - http://sprunge.us/Rf3QgC?en

There were two failed videos ,one of them is 9hrs,not sure why the content creator included it as part of a playlist JavaScript Programming All-in-One Tutorial Series (9 HOURS!) https://www.youtube.com/watch?v=9M4XKi25I2M .
This probably explains why Task output seemed to be hung (failed to download:download appears to be stuck)

The second video that failed https://www.youtube.com/watch?v=NBRdd18N5X0 was also tried individually and still returned a failed output in Task ( failed to download:UNIQUE constraint failed.media.path).

[deldesir]

Happy this is getting more robust. Thanks @EMG70.

The UNIQUE constraint failed.media.path is not difficult to tackle. It's because the stuck videos are not post-timestamped. I'm looking for an elegant way to fix it 🤔.

[holta]

@EMG70 we're hoping a quick test with YouTube Shorts (@deldesir will help) confirms this PR can be merged quickly, ideally today!

[holta]

This PR was tested with #155 and #157 on t4 as explained here:

• Remove residual fragments from failed video #157 (comment)

@deldesir confirms this PR #151 is ready.