Add some unit tests for special tokens

[ClementLokad]

Hello,

Could you add some unit tests to verify that tokens are encoded differently when they are special or not please? Maybe by using the a SpecialTokenMap that changes the default values of XlmRobertaVocab.

[ClementLokad]

Also unskip the unit test TestCreateObject

[EslaMx7]

@ClementLokad Can you please provide more details and examples?

[ClementLokad]

For example, the sentence "Wondering how this will get tokenized 🤔 ?" is not encoded the same way if the following json is passed as the third parameter of XLMRobertaTokenizer

{
  "BosToken": "<s>",
  "ClsToken": "<s>",
  "EosToken": "</s>",
  "MaskToken": "an",
  "PadToken": "<pad>",
  "SepToken": "on",
  "UnkToken": "<unk>"
}

Also, I see in the rust lib https://github.com/guillaume-be/rust-tokenizers/blob/main/main/src/vocab/xlm_roberta_vocab.rs#L219, not all the special token map items (bos_token and mask_token) are added into values compared to here https://github.com/Lokad/Tokenizers/blob/master/src/Lokad.Tokenizers/Vocab/XlmRobertaVocab.cs#L66

[ClementLokad]

Hello Eslam,
Hope you are doing well, did you got the time to look at this issue and the other one?