Afegit scorer amb accents diacrítics i frases del corpus OSCAR

Author: ccoreilly

lm/common-voice/alphabet.txt

@@ -0,0 +1,38 @@
+# Each line in this file represents the Unicode codepoint (UTF-8 encoded)
+# associated with a numeric label.
+# A line that starts with # is a comment. You can escape it with \# if you wish
+# to use '#' as a label.
+ 
+a
+b
+c
+ç
+d
+e
+f
+g
+h
+i
+ï
+j
+k
+l
+m
+n
+o
+p
+q
+r
+s
+t
+u
+ü
+v
+w
+x
+y
+z
+'
+-
+·
+# The last (non-comment) line needs to end with a newline.

lm/ext-diacritics/.gitignore

@@ -0,0 +1,4 @@
+frases.txt
+raw/oscar.txt
+lm.binary
+kenlm.scorer

lm/ext-diacritics/README.md

@@ -0,0 +1,7 @@
+## Scorer extés creat a partir de fonts diverses amb tots els accents diacrítics
+
+- Arxius test, dev i train del dataset de Common Voice (10/12/2019)
+- Frases del dataset Crowdsourced high-quality Catalan speech data set (https://www.openslr.org/69/) 
+- Frases del dataset Ancora a partir del recull d'Universal dependencies (https://github.com/UniversalDependencies/UD_Catalan-AnCora)
+- Frases de la wikipedia recollides pel projecte WikiAnn (https://elisa-ie.github.io/wikiann/)
+- 12.795.447 de frases extretes després de validar i normalitzar el corpus OSCAR (https://oscar-corpus.com/) 

lm/ext-diacritics/alphabet.txt

@@ -0,0 +1,45 @@
+# Each line in this file represents the Unicode codepoint (UTF-8 encoded)
+# associated with a numeric label.
+# A line that starts with # is a comment. You can escape it with \# if you wish
+# to use '#' as a label.
+ 
+a
+à
+b
+c
+ç
+d
+e
+è
+é
+f
+g
+h
+i
+í
+ï
+j
+k
+l
+m
+n
+o
+ò
+ó
+p
+q
+r
+s
+t
+u
+ú
+ü
+v
+w
+x
+y
+z
+'
+-
+·
+# The last (non-comment) line needs to end with a newline.