Problems with finishing project (curration and export)

[yagarea]

Status

Our annotators have finished annotations. Now we have 1500 documents annotated by at least 3 unique annotators. They were annotating document categorization using custom layer (one of inceptions presets) with ability to leave optional note.

• Krippendorff's Alpha (coding / nominal)
  • Mean 0.9113
  • Min 0.0000
  • Max 1.0000
  • Variance 0.0809

How to continue?

We would like to do those things but we do not know how:

• Automatically mark documents with all annotators agreeing with each other as definitively finished/curated.
• Go through documents where is at least one disagreement and authoritatively decide right category.
• Export results (ideally in CSV) in some format that would for every document contain:
  1. Name of document
  2. Final category it got

[reckart]

@yagarea just to understand it correctly. You annotated the documents using a single custom layer of type "document metadata"?

[yagarea]

@reckart
This is our project settings:

[reckart]

The only thing you can do with on-board functionality at the moment is exporting the "Diagnosis" overview from the Agreement page after selecting your label feature. That gives you a CSV table which compares the label values across your annotators and documents. It is the most immediate way of seeing where your annotators disagree. Downside is that you neither see the document text nor the comment in this overview.

We have an experimental auto-curation functionality as well, but it currently does not support document-level annotations. Since we can already do agreement for document-level annotations, the extra work necessary to support them in auto-curation might be small, but I need to look at that.

Proper curation of document-level annotations in the UI is presently not supported. That is definitely quite a bit more work. -> #2702

Another thing that should be possible is to implement a Python script which would read your annotations from a project export and compile document names, labels and comments into a CSV table. It might even already add a flag column indicating if the labels fully agree for a given document. Putting the document text into that CSV would probably be inconvenient unless your documents are very small (i.e. tweets or such). So it would probably make sense to haven an INCEpTION open while going through such a table in order to view the documents on which there was disagreement.

Let me know what you think. I'll ponder a bit more and let you know if I have any other / better idea.

[reckart]

When this is merged, auto-curation of document-level annotation should work.

Still the problem of not being able to see annotator's annotations in any of the curation modes...