Allow importing media to chat

[fabiosangregorio]

It would be awesome if I could send all the media from a chat too. I'd be happy to discuss the feasibility of the implementation with you and putting together a PR for this, after the refactoring issue (#5) is merged.

[ferferga]

Hey, good afternoon! First of all, thank you very much for your interest in improving TLImporter! 😁

Thanks for reminding me about that PR that, in fact, should be closed. The original author didn't commit any other changes and didn't show any sign of activity. In the meanwhile, I added some fixes and merged #8, so I would need to backport those changes to the decomposition of the program.

Aside from that, although I know that my codebase is pretty dirty and should have a better style, I prefer having it this way if it's not going to receive any further message and interest from the people who made those changes, as I already know where each line is located and I'm fully aware of where it's doing something and why. Using object-oriented programming with classes might be easier as well to understand for some people, but in small projects like this, I am more accostumed and I'm faster writing and debugging without objects, as well as I don't really see the necessity of using classes.

I would like first, before doing any further changes, to update the Telethon version used by TLImporter, as I'm currently using the old sync version due to some mandatary changes in the way the Telegram's auth key must be handled by asynchronous processing in Telethon. The new method for reimporting the key makes it not really suitable for its use in the Secret Mode, and, back in the day, I talked with Lonami about ways to simplify and bring back the old behaviour, and, afaik, he did some changes, but didn't want to take all the time again of retesting everything, so I left it as it was. The old version is currently having multiple issues when importing inside 'Saved Messages', so it seems that it's the time to move to the newer version, even if I don't want to.

That's what I have in my roadmap, but I'm not sure when I will be able to do it, as I will be starting exams and practices really soon. But, after that, I think that we can discuss the implementation of media importing. If you do want to help me as well in migrating to the new Telethon version, your help would be greatly appreciated!

Btw, do you have any idea, both convenient/easy for users on how to bring media importing to life? WhatsApp for example doesn't give any info about the media contained in an exported chat, you must decrypt and open the database first if you want to locate the path to each photo/video. That's why I'm interesting in knowing what's on your mind!

Thank you very much again for your interest!

[fabiosangregorio]

Hi, sorry for the delay. I'm starting exams aswell so I don't have much time as of now. :)

I analyzed an export from whatsapp containing some files. Here's the export:

04/09/18, 13:21 - Bea: text text text
04/09/18, 13:22 - Bea: ‎IMG-20180904-WA0002.jpg (file allegato)
text text text
04/09/18, 13:27 - Fabio: text text text
14/12/18, 10:54 - Fabio: ‎PTT-20181214-WA0003.opus (file allegato)

A file is distinguished by "(file allegato)", which in italian means "attached file". Next to that is the file name.

• an image has a file extension of .jpg or .webp
• an audio has a file extension of .opus
  Those files are stored on the user's phone under "Whatsapp > Media" folder. More particularly, we have folders such as "Whatsapp Images", "Whatsapp Audio", etc ...

For importing files to the Telegram chat we could ask the user to save the Whatsapp folder to his computer and provide us with the path. We could then match the filename in the txt export with the filename, and the extension in the txt with the folder, and send it to the telegram chat. Could this work?

[ferferga]

@fabiosangregorio I didn't reply you back in the day when you posted this, but I checked it and you're right, media paths are specified in Whatsapp exports. I only exported chats where I had all the media removed from my storage, so it appeared in the txt as <media ommitted>. So your idea seems feasible.

I finished adapting TLImporter to the new telethon versions (and Telegram layers), new secret mode is now live and all the work I wanted to do with the file parsing logic is done in TLImporter 3.0.6.

I won't release any new updates (if not new major bugs appear), as the app is pretty mature now and it covers even more that what I needed when I decided to make it. So, you have plenty of time to make a PR if you want to to add this feature, you probably won't need to deal with merge conflicts :).

I don't need to import media (and I probably won't in the future), so I don't really have the motivation to make it real on my own, but any PR willing to improve the tools is welcome, so if you want to make it, I encourage you to do so and I thank you very much for investing your time :). It's nice to see that people likes your work and want to improve it, so thanks!

Some difficulties that come up to my mind that you might need to solve for getting this done:

• Path conversion from the Android structure, making it somehwat flexible so it works right if WhatsApp makes minor changes to them.
• Correct path handling both for Windows and Unix systems (this should be somewhat easy with pathlib, or by replacing the \ in Windows to /, like I do right now because I didn't know about pathlib back in the day.
• (file allegato) is different in each language, so we don't want to depend on it for parsing the files: in fact, we wouldn't want that to be imported, it's better if we ignore it.
• Detect that something is media with the wide variety of files that are out there. You don't only want to import videos, voice notes and photos, also documents! Maybe you can try to make some kind of regex with dots . and check with path.isfile if the file is present in the same folder where the TLImporter script is present?

I think that using path.isfile with a wide variety of regexes and a correct string splitting might make this easier and also avoid false positives.

[eutampieri]

Hey! I was working on a WhatsApp export parser too! @fabiosangregorio are you Italian? If so, my fields vary a little from yours. Anyway, if you want some help let me know

[fabiosangregorio]

Sorry, as always too many projects and too little time. I won't be able to put together a PR for a long time. University is time consuming :)
Yes, the export language is indeed italian.

[eutampieri]

@fabiosangregorio is your export from WhatsApp? Mine (from Android and iOS) are formatted differently…
They’re formatted like this:
[timestamp] sender: message
With message being either the text or the <attachment(allegato): filename> for attachments

[fabiosangregorio]

Exporting a chat via Chat > Menu > Other > Export chat > Share via... > Email results in two files being sent via email:

Chat txt file

The chat file is formatted as

DD/MM/YY, HH:mm - SenderName SenderSurname: [attachment.ext (attached file)] \n Chat text or file caption

with [] being optional.

Chat WhatsApp con Lucia RedactedSurname.txt

17/03/18, 18:54 - I messaggi inviati a questa chat e le chiamate sono ora protetti con la crittografia end-to-end. Tocca per maggiori info. 17/03/18, 18:54 - Lucia RedactedSurname: RedactedText 08/05/18, 18:27 - Fabio: RedactedText 08/05/18, 18:43 - Lucia RedactedSurname: RedactedText 👍🏼🤪 08/05/18, 22:09 - Fabio: RedactedText 08/05/18, 22:09 - Fabio: RedactedText 08/05/18, 22:10 - Lucia RedactedSurname: RedactedText 08/05/18, 22:38 - Fabio: RedactedText 22/11/18, 08:27 - Lucia RedactedSurname: ‎IMG-20181122-WA0000.jpg (file allegato) RedactedText 👋🏻👋🏻 // This is the caption to the previous image 22/11/18, 08:35 - Fabio: RedactedText 24/12/18, 08:58 - Lucia RedactedSurname: RedactedText 24/12/18, 09:55 - Fabio: RedactedText 😋 27/10/19, 09:53 - Lucia RedactedSurname: RedactedText 27/10/19, 09:55 - Fabio: RedactedText 01/11/19, 19:37 - Lucia RedactedSurname: RedactedText 01/11/19, 19:45 - Fabio: RedactedText

Image file

IMG-20181122-WA0000.jpg

[eutampieri]

_chat 2.txt
Here's mine

[eutampieri]

@ferferga can I implement an alternative parser which gets activated if the program finds that messages are in the format above? Because otherwise the software imports messages as an unique looong message. Maybe I should open another issue instead to keep track of this...

[ferferga]

@eutampieri I think this is the same problem as the one reported in #11. I'm looking into it.

[eutampieri]

Yes, it’s the same issue but it doesn’t crash

[eutampieri]

I'm starting to have @fabiosangregorio's problem…. I don't have an ETA yet