Is the encoding attribute supported on deserialization?

[sschuberth]

I have an XML document with

<?xml version="1.0" encoding="ISO-8859-1" ?>

and it contains strings with German Umlauts like

<LigaTitel>Länderspiele - Junioren U16</LigaTitel>

which seem to be returned as "L�nderspiele - Junioren U16". Is there anything I need to configure explicitly to make this work correctly?

[pdvrieze]

The support for encoding depends very much on a number of factors. The first one is that is only supported on InputStream instances (Reader/String should already be decoded). Another one is that it only directly works with the Java/Jvm platform xml readers (note that this uses service loaders on the JVM side, so can be adjusted, for example by loading ). The platform independent parser (KtXmlReader) doesn't support InputStream, only Reader.

So, it should work on the JVM/Android platforms when using InputStreams where the underlying StaxParser implementation supports encodings.
Note that the native InputStreamReader only works with UTF-8 (for now) - if I recall correctly there is a version in the xmlschema branch that handles detecting UTF-16LE/UTF-16BE byte order marks.

[sschuberth]

So, it should work on the JVM/Android platforms when using InputStreams where the underlying StaxParser implementation supports encodings.

I'm currently building for JBM only, and I use code like

val leaguesXml = javaClass.getResource("/HED-Ligen.xml").readText()
val leagues = xml.decodeFromString<HockeyLigen>(leaguesXml)

where

val xml = XML {
    policy = object : DefaultXmlSerializationPolicy({ pedantic = false }) {
        @ExperimentalXmlUtilApi
        override fun defaultOutputKind(serialKind: SerialKind) =
            OutputKind.Element // All values are encoded as elements (no attributes are used).
    }
}

So that case should be supported, right?

PS: I've verified that the XML file actually is encoded as specified by lookign at IntelliJ's detected encoding, see

[sschuberth]

Ah, wait, it's probably my fault: readText() assumes UTF-8, I guess! I'll check.

[sschuberth]

Yes, it works with readText(Charsets.ISO_8859_1), sorry for that. But is there a way to let xmlutil determine the encoding when reading the file from a resource / InputStream on JVM? As there is no decodeFromStream I guess I should use decodeFromReader and construct an XmlReader somehow?

[sschuberth]

Ok, this works for me:

val leaguesResources = javaClass.getResource("/HED-Ligen.xml")

val leagues = leaguesResources.openStream().use {
    xml.decodeFromReader<HockeyLigen>(StAXReader(it, encoding = null))
}

Edit: Apparently passing encoding = null causes the encoding to be detected correctly. Would you accept a PR that makes this the default value for that parameter, @pdvrieze?

[pdvrieze]

Yes, readText would do that, it can not do anything else. Supporting character decoding to be determined by the content (the xml declaration) is not trivial and must be directly supported by the decoder (as it needs to change the decoding "on the fly").

[pdvrieze]

@sschuberth I'd need to check whether it would be a breaking API change to change it. But it should be the default absent that being an issue.

[sschuberth]

See #206.

[pdvrieze]

I've just pushed a slightly different approach (the source API should be (mostly) the same), and thank you very much for the pull request