Nuno Guimarães, Purificação Silvano, Ricardo Campos, Alipio Jorge, Ana Filipa Pacheco, Dimitar Iliyanov Dimitrov, Nikolaos Nikolaidis, Roman Yangarber, Elisa Sartori, Nicolas Stefanovitch, Preslav Nakov, Jakub Piskorski, Giovanni Da San Martino
NarratEX Dataset: Explaining the Dominant Narratives in News Texts
can be accessed and used for research purposes in the following manner:
-
Please register to the Task 10 at SemEval 2025 at: https://propaganda.math.unipd.it/semeval2025task10/
-
Once registered and vetted the access will be granted to train and dev data of the shared task which corresponds to TRAIN and TEST the dataset described in the paper.
TRAIN set: The TRAIN set from the paper corresponds to the union of the files: batch 1-3 (Dec. 4) and "Additional Training data for Russian" File names after downloading: target_4_December_release.zip and training_data_RU_final_19_January_2025_release.zip respectively.
TEST set: Display name on website: Development set File name after downloading: target_4_December_release.zip
TBA
> @inproceedings{tba}
The following table presents the results for the generative baselines of the narrative inference classification task mentioned in the paper applied to the entirety of the dataset.
| Portuguese | |||||
|---|---|---|---|---|---|
| model | narrative | precision | recall | f1 | support |
| gemini | Amplifying Climate Fears | 0.630 | 0.239 | 0.347 | 71 |
| gemini | Amplifying war-related fears | 0.059 | 0.043 | 0.050 | 23 |
| gemini | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 10 |
| gemini | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Criticism of climate movement | 0.000 | 0.000 | 0.000 | 7 |
| gemini | Criticism of climate policies | 0.111 | 0.400 | 0.174 | 5 |
| gemini | Criticism of institutions and authorities | 0.125 | 0.040 | 0.061 | 25 |
| gemini | Discrediting Ukraine | 0.267 | 0.077 | 0.119 | 52 |
| gemini | Discrediting the West, Diplomacy | 0.250 | 0.074 | 0.114 | 27 |
| gemini | Distrust towards Media | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Downplaying climate change | 0.000 | 0.000 | 0.000 | 8 |
| gemini | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 2 |
| gemini | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 3 |
| gemini | Praise of Russia | 0.000 | 0.000 | 0.000 | 23 |
| gemini | Russia is the Victim | 0.250 | 0.231 | 0.240 | 13 |
| gemini | Speculating war outcomes | 0.083 | 0.200 | 0.118 | 5 |
| lamma3 | Amplifying Climate Fears | 0.593 | 0.225 | 0.327 | 71 |
| lamma3 | Amplifying war-related fears | 0.053 | 0.043 | 0.048 | 23 |
| lamma3 | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 10 |
| lamma3 | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Criticism of climate movement | 0.000 | 0.000 | 0.000 | 7 |
| lamma3 | Criticism of climate policies | 0.111 | 0.400 | 0.174 | 5 |
| lamma3 | Criticism of institutions and authorities | 0.125 | 0.040 | 0.061 | 25 |
| lamma3 | Discrediting Ukraine | 0.200 | 0.058 | 0.090 | 52 |
| lamma3 | Discrediting the West, Diplomacy | 0.300 | 0.111 | 0.162 | 27 |
| lamma3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Downplaying climate change | 0.000 | 0.000 | 0.000 | 8 |
| lamma3 | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 2 |
| lamma3 | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 3 |
| lamma3 | Praise of Russia | 0.000 | 0.000 | 0.000 | 23 |
| lamma3 | Russia is the Victim | 0.182 | 0.154 | 0.167 | 13 |
| lamma3 | Speculating war outcomes | 0.083 | 0.200 | 0.118 | 5 |
| phi3 | Amplifying Climate Fears | 0.600 | 0.211 | 0.313 | 71 |
| phi3 | Amplifying war-related fears | 0.059 | 0.043 | 0.050 | 23 |
| phi3 | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 10 |
| phi3 | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Criticism of climate movement | 0.000 | 0.000 | 0.000 | 7 |
| phi3 | Criticism of climate policies | 0.118 | 0.400 | 0.182 | 5 |
| phi3 | Criticism of institutions and authorities | 0.250 | 0.120 | 0.162 | 25 |
| phi3 | Discrediting Ukraine | 0.267 | 0.077 | 0.119 | 52 |
| phi3 | Discrediting the West, Diplomacy | 0.200 | 0.074 | 0.108 | 27 |
| phi3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Downplaying climate change | 0.000 | 0.000 | 0.000 | 8 |
| phi3 | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 2 |
| phi3 | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 3 |
| phi3 | Praise of Russia | 0.000 | 0.000 | 0.000 | 23 |
| phi3 | Russia is the Victim | 0.182 | 0.154 | 0.167 | 13 |
| phi3 | Speculating war outcomes | 0.083 | 0.200 | 0.118 | 5 |
| English | |||||
| model | narrative | precision | recall | f1 | support |
| gemini | Amplifying Climate Fears | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Amplifying war-related fears | 0.091 | 0.050 | 0.065 | 20 |
| gemini | Blaming the war on others rather than the invader | 0.250 | 0.273 | 0.261 | 11 |
| gemini | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Controversy about green technologies | 0.091 | 0.167 | 0.118 | 6 |
| gemini | Criticism of climate movement | 0.308 | 0.160 | 0.211 | 25 |
| gemini | Criticism of climate policies | 0.231 | 0.176 | 0.200 | 17 |
| gemini | Criticism of institutions and authorities | 0.333 | 0.115 | 0.171 | 26 |
| gemini | Discrediting Ukraine | 0.364 | 0.235 | 0.286 | 17 |
| gemini | Discrediting the West, Diplomacy | 0.125 | 0.031 | 0.050 | 32 |
| gemini | Distrust towards Media | 0.000 | 0.000 | 0.000 | 7 |
| gemini | Downplaying climate change | 0.077 | 0.125 | 0.095 | 8 |
| gemini | Green policies are geopolitical instruments | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Hidden plots by secret schemes of powerful groups | 0.100 | 0.083 | 0.091 | 24 |
| gemini | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 7 |
| gemini | Overpraising the West | 0.000 | 0.000 | 0.000 | 4 |
| gemini | Praise of Russia | 0.000 | 0.000 | 0.000 | 5 |
| gemini | Questioning the measurements and science | 0.091 | 0.111 | 0.100 | 9 |
| gemini | Russia is the Victim | 0.000 | 0.000 | 0.000 | 3 |
| gemini | Speculating war outcomes | 0.167 | 0.222 | 0.190 | 9 |
| lamma3 | Amplifying Climate Fears | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Amplifying war-related fears | 0.083 | 0.050 | 0.063 | 20 |
| lamma3 | Blaming the war on others rather than the invader | 0.250 | 0.273 | 0.261 | 11 |
| lamma3 | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Controversy about green technologies | 0.091 | 0.167 | 0.118 | 6 |
| lamma3 | Criticism of climate movement | 0.333 | 0.200 | 0.250 | 25 |
| lamma3 | Criticism of climate policies | 0.200 | 0.118 | 0.148 | 17 |
| lamma3 | Criticism of institutions and authorities | 0.250 | 0.077 | 0.118 | 26 |
| lamma3 | Discrediting Ukraine | 0.250 | 0.118 | 0.160 | 17 |
| lamma3 | Discrediting the West, Diplomacy | 0.300 | 0.094 | 0.143 | 32 |
| lamma3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 7 |
| lamma3 | Downplaying climate change | 0.091 | 0.125 | 0.105 | 8 |
| lamma3 | Green policies are geopolitical instruments | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Hidden plots by secret schemes of powerful groups | 0.105 | 0.083 | 0.093 | 24 |
| lamma3 | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 7 |
| lamma3 | Overpraising the West | 0.000 | 0.000 | 0.000 | 4 |
| lamma3 | Praise of Russia | 0.000 | 0.000 | 0.000 | 5 |
| lamma3 | Questioning the measurements and science | 0.182 | 0.222 | 0.200 | 9 |
| lamma3 | Russia is the Victim | 0.000 | 0.000 | 0.000 | 3 |
| lamma3 | Speculating war outcomes | 0.167 | 0.222 | 0.190 | 9 |
| phi3 | Amplifying Climate Fears | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Amplifying war-related fears | 0.091 | 0.050 | 0.065 | 20 |
| phi3 | Blaming the war on others rather than the invader | 0.250 | 0.273 | 0.261 | 11 |
| phi3 | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 6 |
| phi3 | Criticism of climate movement | 0.357 | 0.200 | 0.256 | 25 |
| phi3 | Criticism of climate policies | 0.273 | 0.176 | 0.214 | 17 |
| phi3 | Criticism of institutions and authorities | 0.400 | 0.154 | 0.222 | 26 |
| phi3 | Discrediting Ukraine | 0.222 | 0.118 | 0.154 | 17 |
| phi3 | Discrediting the West, Diplomacy | 0.333 | 0.125 | 0.182 | 32 |
| phi3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 7 |
| phi3 | Downplaying climate change | 0.071 | 0.125 | 0.091 | 8 |
| phi3 | Green policies are geopolitical instruments | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Hidden plots by secret schemes of powerful groups | 0.059 | 0.042 | 0.049 | 24 |
| phi3 | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 7 |
| phi3 | Overpraising the West | 0.000 | 0.000 | 0.000 | 4 |
| phi3 | Praise of Russia | 0.000 | 0.000 | 0.000 | 5 |
| phi3 | Questioning the measurements and science | 0.100 | 0.111 | 0.105 | 9 |
| phi3 | Russia is the Victim | 0.000 | 0.000 | 0.000 | 3 |
| phi3 | Speculating war outcomes | 0.091 | 0.111 | 0.100 | 9 |
| Russian | |||||
| model | narrative | precision | recall | f1 | support |
| gemini | Amplifying war-related fears | 0.167 | 0.250 | 0.200 | 4 |
| gemini | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 10 |
| gemini | Discrediting Ukraine | 0.900 | 0.153 | 0.261 | 59 |
| gemini | Discrediting the West, Diplomacy | 0.167 | 0.045 | 0.071 | 22 |
| gemini | Distrust towards Media | 0.000 | 0.000 | 0.000 | 2 |
| gemini | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 3 |
| gemini | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Praise of Russia | 0.750 | 0.196 | 0.310 | 46 |
| gemini | Russia is the Victim | 0.000 | 0.000 | 0.000 | 12 |
| gemini | Speculating war outcomes | 0.200 | 0.500 | 0.286 | 2 |
| lamma3 | Amplifying war-related fears | 0.000 | 0.000 | 0.000 | 4 |
| lamma3 | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 10 |
| lamma3 | Discrediting Ukraine | 0.875 | 0.119 | 0.209 | 59 |
| lamma3 | Discrediting the West, Diplomacy | 0.125 | 0.045 | 0.067 | 22 |
| lamma3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 2 |
| lamma3 | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 3 |
| lamma3 | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Praise of Russia | 0.625 | 0.109 | 0.185 | 46 |
| lamma3 | Russia is the Victim | 0.000 | 0.000 | 0.000 | 12 |
| lamma3 | Speculating war outcomes | 0.200 | 0.500 | 0.286 | 2 |
| phi3 | Amplifying war-related fears | 0.000 | 0.000 | 0.000 | 4 |
| phi3 | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 10 |
| phi3 | Discrediting Ukraine | 0.750 | 0.153 | 0.254 | 59 |
| phi3 | Discrediting the West, Diplomacy | 0.200 | 0.091 | 0.125 | 22 |
| phi3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 2 |
| phi3 | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 3 |
| phi3 | Negative Consequences for the West | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Praise of Russia | 0.727 | 0.174 | 0.281 | 46 |
| phi3 | Russia is the Victim | 0.000 | 0.000 | 0.000 | 12 |
| phi3 | Speculating war outcomes | 0.000 | 0.000 | 0.000 | 2 |
| Bulgarian | |||||
| model | narrative | precision | recall | f1 | support |
| gemini | Amplifying Climate Fears | 0.500 | 0.160 | 0.243 | 81 |
| gemini | Amplifying war-related fears | 0.263 | 0.106 | 0.152 | 47 |
| gemini | Blaming the war on others rather than the invader | 0.067 | 0.043 | 0.053 | 23 |
| gemini | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Criticism of climate movement | 0.053 | 1.000 | 0.100 | 1 |
| gemini | Criticism of climate policies | 0.000 | 0.000 | 0.000 | 13 |
| gemini | Criticism of institutions and authorities | 0.048 | 0.111 | 0.067 | 9 |
| gemini | Discrediting Ukraine | 0.250 | 0.060 | 0.096 | 67 |
| gemini | Discrediting the West, Diplomacy | 0.294 | 0.077 | 0.122 | 65 |
| gemini | Distrust towards Media | 0.000 | 0.000 | 0.000 | 3 |
| gemini | Downplaying climate change | 0.038 | 0.111 | 0.057 | 9 |
| gemini | Green policies are geopolitical instruments | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Hidden plots by secret schemes of powerful groups | 0.029 | 0.250 | 0.051 | 4 |
| gemini | Negative Consequences for the West | 0.045 | 0.125 | 0.067 | 8 |
| gemini | Overpraising the West | 0.000 | 0.000 | 0.000 | 1 |
| gemini | Praise of Russia | 0.111 | 0.040 | 0.059 | 25 |
| gemini | Russia is the Victim | 0.000 | 0.000 | 0.000 | 8 |
| gemini | Speculating war outcomes | 0.000 | 0.000 | 0.000 | 18 |
| lamma3 | Amplifying Climate Fears | 0.481 | 0.160 | 0.241 | 81 |
| lamma3 | Amplifying war-related fears | 0.238 | 0.106 | 0.147 | 47 |
| lamma3 | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 23 |
| lamma3 | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Criticism of climate movement | 0.053 | 1.000 | 0.100 | 1 |
| lamma3 | Criticism of climate policies | 0.000 | 0.000 | 0.000 | 13 |
| lamma3 | Criticism of institutions and authorities | 0.048 | 0.111 | 0.067 | 9 |
| lamma3 | Discrediting Ukraine | 0.188 | 0.045 | 0.072 | 67 |
| lamma3 | Discrediting the West, Diplomacy | 0.278 | 0.077 | 0.120 | 65 |
| lamma3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 3 |
| lamma3 | Downplaying climate change | 0.000 | 0.000 | 0.000 | 9 |
| lamma3 | Green policies are geopolitical instruments | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Hidden plots by secret schemes of powerful groups | 0.029 | 0.250 | 0.051 | 4 |
| lamma3 | Negative Consequences for the West | 0.048 | 0.125 | 0.069 | 8 |
| lamma3 | Overpraising the West | 0.000 | 0.000 | 0.000 | 1 |
| lamma3 | Praise of Russia | 0.111 | 0.040 | 0.059 | 25 |
| lamma3 | Russia is the Victim | 0.000 | 0.000 | 0.000 | 8 |
| lamma3 | Speculating war outcomes | 0.000 | 0.000 | 0.000 | 18 |
| phi3 | Amplifying Climate Fears | 0.458 | 0.136 | 0.210 | 81 |
| phi3 | Amplifying war-related fears | 0.176 | 0.064 | 0.094 | 47 |
| phi3 | Blaming the war on others rather than the invader | 0.000 | 0.000 | 0.000 | 23 |
| phi3 | Climate change is beneficial | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Controversy about green technologies | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Criticism of climate movement | 0.053 | 1.000 | 0.100 | 1 |
| phi3 | Criticism of climate policies | 0.000 | 0.000 | 0.000 | 13 |
| phi3 | Criticism of institutions and authorities | 0.037 | 0.111 | 0.056 | 9 |
| phi3 | Discrediting Ukraine | 0.133 | 0.030 | 0.049 | 67 |
| phi3 | Discrediting the West, Diplomacy | 0.286 | 0.092 | 0.140 | 65 |
| phi3 | Distrust towards Media | 0.000 | 0.000 | 0.000 | 3 |
| phi3 | Downplaying climate change | 0.074 | 0.222 | 0.111 | 9 |
| phi3 | Green policies are geopolitical instruments | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Hidden plots by secret schemes of powerful groups | 0.000 | 0.000 | 0.000 | 4 |
| phi3 | Negative Consequences for the West | 0.048 | 0.125 | 0.069 | 8 |
| phi3 | Overpraising the West | 0.000 | 0.000 | 0.000 | 1 |
| phi3 | Praise of Russia | 0.111 | 0.040 | 0.059 | 25 |
| phi3 | Russia is the Victim | 0.000 | 0.000 | 0.000 | 8 |
| phi3 | Speculating war outcomes | 0.000 | 0.000 | 0.000 | 18 |