Table build schema evolution

[marcos-lg]

When the table build reads the avro files it takes the schema of the first file it reads. If new fields are added they won't be considered if they are not in the first file it reads.

Spark options such as schemaMerge or passing the schema didn't seem to work in some tests I did.

[timrobertson100]

I'm a bit rusty on this, but I think back in the day it was common to keep a schema registry (e.g. confluent schema registry) and then as long as you followed predictable patterns of adding fields, but not changing or removing existing, you could state the latest schema when reading. e.g. spark.read.format("avro").schema(<your struct type here>).load(...).

Would stating the schema you want be an option perhaps?

[marcos-lg]

I tried doing that but didn't seem to work (I can revisit it again later).