数据集 (About the dataset)

[yurui12138]

您好，我想问一下程序中‘data_6_remove_dup_train.csv’这样的数据文件是如何得到的？您这边方便提供吗？或者您方便告诉一下这个数据文件的结构吗？

[chenyangh]

Hi, from Jiwei's GitHub page (https://github.com/jiweil/Neural-Dialogue-Generation), you can download his data file s_given_t_dialogue_length2_6.txt. That is the original source, however, there are about 40% (if I remember correctly) of the data are duplicated. So you need to remove them.

Each line in the data_6_remove_dup_train.csv is supposed to be in the format of
source \t target \t emotion \n .

Emotion can be generated by any reasonably good classifier that is trained on CBET dataset. I was using some simple bi-LSTM, and we have much better tools now.

[yurui12138]

Thank you very much for your reply.

[Ellen217]

I have read your reply above.
I followed what was posted on Jiwei's GitHub page, but did not get 'data_6_remove_dup_train.csv'.

If possible, could you send the file to my email?
This is 4threvolution1@gmail.com.
Please..
I'll be waiting for the reply.

[chenyangh]

Hi, unfortunately, I did not keep files. However, this file can be easily generated by three steps:

• First, you need to replace the IDs in Jiwei's data (with minimum length 6) with real words. There is an ID to word mapping file, which you can find it here.

• Removing duplications: there were about 40% of lines are duplicated. I did this by Linux commend, sort followed by and uniq.

• Last but not least, you need to train an emotion classifier then run inference over the sentences, to label the emotions automatically. I trained a bi-LSTM on CBET dataset in 2017, and there are much better options at this moment

Thanks for being interested in this work. I hope my answer can help.

[chenyangh]

@CHI5E I have found the data and uploaded them to google drive.
https://drive.google.com/drive/folders/1TiHxaajx2w0Cdz3FjD6PqSjy6qM0IUTD?usp=sharing

Cheers.

[Ellen217]

Thank you so much for your quick feedback!!!!! I will study hard! 👍 2020년 3월 15일 (일) 오전 10:13, Chenyang Huang <notifications@github.com>님이 작성:

…

Reopened #3 <#3>. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#3 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALROPW2GIXECQXVKYNTWQGDRHQTTHANCNFSM4JNHS75A> .

[Ellen217]

Hello!
The file you uploaded to Google Drive was very well received and trained all the models.
But I still don't understand the data='data_6_remove_dup_train.csv'.
The following are my questions.

1. The result of Sequence Transduction on 'Source' is 'Target'?
   I wonder if another meaning of 'source' is 'target'.
   I thought 'target' was the answer to source.

2. Is 'tag' a result classified in CBET.csv?

3. I wonder what the original data structure was for 'data_6_remove_dup_train.csv'.
   The data seems to be well alignment between sentences.

Thank you for the good paper and I look forward to your answer.
Thank you very much!

[chenyangh]

Hello @Chl5e ,

1. Yes, they are usually paired sequences, e.g. in machine translation, they can be English and French. There might be other notations, I use source and target to refer to that a model is trying to generate the target, given source.

2. Yes, the corresponding emotion which is estimated by any reasonable classifier.

3. They are from the Opensubtitle dataset, which are movie transcripts. The original preprocessing is done by Jiwei, I further remove duplicates.

[Ellen217]

Thank you so much for your answer.
Thanks to you, I'm getting more understanding!
Then let me ask you one more question!
In this paper, I saw that you used a pre-trained 'FastText' Model.
Is it https://fasttext.cc/docs/en/english-vectors.html?
if you's

1. wiki-news-300d-1M.vec.zip: 1 million word vectors trained on Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset (16B tokens).
2. wiki-news-300d-1M-subword.vec.zip: 1 million word vectors trained with subword infomation on Wikipedia 2017, UMBC webbase corpus and statmt.org news dataset (16B tokens).
3. crawl-300d-2M.vec.zip: 2 million word vectors trained on Common Crawl (600B tokens).
4. crawl-300d-2M-subword.zip: 2 million word vectors trained with subword information on Common Crawl (600B tokens).

Which model did you put in the directory 'feature' and used?
Thank you always!..!ㅠㅠ

[chenyangh]

@Chl5e Hi, I think you can use any pre-trained word vector models, including glove and google word2vec. The choice of word vector models should not affect the results by a large margin. I can't remember which exact model I used at this moment. In my later experiments, glove performs better than fasttext if not using subword representation.