Running SLTev on ESIC (interpretation corpus)

[bhaddow]

Hi

We are trying to test text-to-text translation on the ESIC corpus. The problem is that ESIC is document aligned, but not sentence aligned. The documents are segmented, but the number of segments does not match between source and target, so SLTev throws an error. Yet it states in the documentation that "segmentation can differ from the reference one"

How should this case be handled in SLTev?

evaluation for  en.OSt.man.orto.txt.slt  failed, the number of Complete lines (C) in  en.OSt.man.orto.txt.OStt  and  en.IStde.man.orto.txt  are not equal
The number of C segment (complete) in en.OSt.man.orto.txt.OStt is 2693 and number of lines in en.IStde.man.orto.txt is 2900

best
Barry

[mohammad2928]

Hi Barry,

when you want to evaluate SLT files (MT with timestamps), you need to OStt (OSt with timestamps) and reference. So, the number of complete segments in OStt and reference must be equal.

Yet it states in the documentation that "segmentation can differ from the reference one"

Yes, it means candidates (MT, SLT, ...) can have a different segmentation from the reference. But, in this case, OStt and reference have a different segmentation and it is not correct.

Please notice that OStt and reference files are gold.

If possible, please share with me some of your evaluation files for more help.

Thanks,
Mohammad

[bhaddow]

Hi Mohammed

Do the OStt and reference files need to match for the latency and flicker calculation? Because bleu could use docAsWhole.

I'm attaching the files below (I have to give them txt extensions)

ref en.IStde.man.orto.txt

Ostt en.OSt.man.orto.txt.OStt.txt

slt en.OSt.man.orto.txt.slt.txt

best
Barry

[mohammad2928]

Hi,

There are two ways for solving this problem:
1- The first one is to convert SLT to MT and then using MTeval.
2- The second one is to calculate the BLEU score and flicker for these types of files (the number of complete segments in reference and OStt are not equal.) (If need I will update SLTev to support these files)

what do you think about them? which one is better?
If you have an idea please share it with me.

Also, I am going to add some scripts and the main points to convert candidate types to each other.

For example, I am going to add the following main points:
SLTtoMT (convert SLT to MT )
MTtoSLT (convert MT to SLT )
ASRTtoASR (convert ASRT to ASR)
ASRtoASRT (convert ASR to ASRT)

What do you think about it? Are they useful?

Best,
Mohammad

[bhaddow]

Hi Mohammed

So for solution 1, this would just evaluate the output as MT? If we converted each document to a single segment then this would work, since the corpus is document aligned. But we could then just do this directly with sacrebleu?

I think 2 is more useful, but that means we have to solve how to define flicker and latency when the OStt C-segments and the reference segments do not match - is that correct? I am not sure what the solution is, I would have to think about.

For the conversion tools, is the MT to SLT tool similar to what I asked about in #33 ? Yes, that would be useful. I think @sukantasen has a script for that. For the other direction, it's less important.

best
Barry

[mohammad2928]

Hi Barry,

[mohammad2928]

Hi Barry,

So for solution 1, this would just evaluate the output as MT? If we converted each document to a single segment then this > would work, since the corpus is document aligned. But we could then just do this directly with sacrebleu?

I think 2 is more useful, but that means we have to solve how to define flicker and latency when the OStt C-segments and the reference segments do not match - is that correct? I am not sure what the solution is, I would have to think about.

I have updated SLTev for solving this issue, you can upgrade it to the version v1.2. 2 (pip install --upgrade SLTev)
When the number of complete segments in the OStt and reference is not equal, just Bleuscore (docsAsWhole and MWERsegmenter) and Flicker will be calculated.

For the conversion tools, is the MT to SLT tool similar to what I asked about in #33 ? Yes, that would be useful. I think @sukantasen has a script for that. For the other direction, it's less important.

I will add them in the next version.

Best,
Mohammad

[bhaddow]

Hi Mohammed

That sounds good!

For "docAsWhole", does that mean that the whole test corpus is treated as a document? Is there any way to tell SLTev that the corpus is made up of a number of smaller documents?

best
Barry

[mohammad2928]

Hi Barry,

For "docAsWhole", does that mean that the whole test corpus is treated as a document?

Yes, it concatenates whole test corpus segments as a document.

Is there any way to tell SLTev that the corpus is made up of a number of smaller documents?

I do not understand your meaning exactly! But in the second BleuScore calculation (using MWERsegmenter), it uses MWERsegmenter to resegment candidate segments according to the reference segments. But there is no way to tell SLTev that the corpus is made of multiple documents.
If you have any idea for that please share with me.

Best,
Mohammad

[bhaddow]

But there is no way to tell SLTev that the corpus is made of multiple documents.

This is what I thought.
We could allow the user to mark them somehow, for example leaving a blank line between the documents in the reference file? Or we could add an extra letter to the C/P annotation in the source?

[mohammad2928]

I think using blank line make SLTev complex.
Could you please explain what is the purpose of the multi-document files?

I think you are going to get scores for each document seperatly. Yes?
I think the best way is making an special token (e.g. ###ENDDOCUMNET###) and making a parameter (e.g. --multi) to define the types of documents for evaluation. But the main question is that how we should treat with the multi-document files.

proposed idea:
1- Making a parser to pars multi-documents and split them to the multiple documents.
Note: Reference, candidate(e.g. SLT) and ostt must be multi-documents (contain defined token).
2- evaluating splited documents speratly

For example, suppose there are 3 files test.slt, test.ostt and test.ref and test.ref contains 3 documents(e.g. [1,2,3]).
each file contains two defined tokens.

step1:
parse files per number of documents. So there will be the following files:
[test.1.slt, test.2.slt, test.3.slt]
[test.1.ref, test.2.ref, test.3.ref]
[test.1.ostt, test.2.ostt, test.3.ostt]

step2:
The following evaluations run seperatly.
SLTeval -i test.1.slt test.1.ref test.1.ostt -f slt ref ostt
SLTeval -i test.2.slt test.2.ref test.2.ostt -f slt ref ostt
SLTeval -i test.3.slt test.3.ref test.3.ostt -f slt ref ostt

What do you think about this idea?

[bhaddow]

Hi Mohammed

The dev set of the ESIC corpus is aligned at the document level, but not at the sentence level. It contains 28 documents. It's a corpus of interpretation, so should be ideal for evaluating simultaneous SLT, and so we want to use SLTev.

Evaluating all 28 documents separately seems quite awkward for the user. Adding a special token between documents could work.

best
Barry

[mohammad2928]

Hi Barry,

Could you please share with me an example of the dev set?

Evaluating all 28 documents separately seems quite awkward for the user.

What scores would you expect to evaluate?
calculating BleuScore and Flicker are ok but the Delay is complex.

The next idea (for calculating quality score):
1- Concatenating all document sentences as a sentence (in candidate and reference).
2- Calculating the BleuScore
Note: we this method we can use MWERsegmenter to resegment candidate segments.
Do you agree with that?

For example, if there is a file with 28 documents.
we will have a file with 28 lines.

Best,
Mohammad

[bhaddow]

Hi Mohammed

I linked to a document above, and it's from the ESIC corpus https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3719

For example, if there is a file with 28 documents.
we will have a file with 28 lines.

Yes, this seems fine. We could prepare the data this way, or SLTev could convert it internally. But I am not sure why calculating delay is complex? Each of the 28 documents is a speech, and there is an ostt file with timestamps for each speech

best
Barry

[mohammad2928]

Hi Barry,

I linked to a document above, and it's from the ESIC corpus https://lindat.mff.cuni.cz/repository/xmlui/handle/11234/1-3719

Thanks.

Yes, this seems fine. We could prepare the data this way, or SLTev could convert it internally. But I am not sure why calculating delay is complex? Each of the 28 documents is a speech, and there is an ostt file with timestamps for each speech

SLTev will convert them internally.
Delay calculation will be complex if we want to calculate the delay for each document separately. For example, if we have a file with 20 documents, there is a need to calculate all types of delays for each document. But if we want to calculate delay once for a file (with several documents) it is ok.

Let's summarise the above messages and if you agree with them please confirm and, I will start to implement them.

A. Adding a file converter to the SLTev

1. SLTtoREF (for convert SLT/ASRT files to reference/ASR files)
   Note: conversely, it is not useful, because delay and flicker are zero for MT and ASR evaluation.

B. Adding multi-document support module

1. Adding a token to separate documents in the OStt, reference, and candidates (e.g. ###ENDDOCUMNET###).
2. Removing the separation token and calculation delay and flicker as normally.
3. For quality scores, concatenating all document sentences as a sentence (in candidate and reference).

Thanks,
Mohammad

[bhaddow]

Hi Mohammad

This sounds good. Just a couple of questions:

SLTtoREF (for convert SLT/ASRT files to reference/ASR files)

What is this needed for?

Removing the separation token and calculation delay and flicker as normally.

In our case, the timestamps start from 0 for each document, so will this work as normal?

best
Barry