Can't clear color values

If you have an optional color field  and set a color value (hex number), the field can no longer be empty.

For exampleI set the field to white (#ffffff) and save. I then click "clear" and save. Value is still white. If I delete the value from the field manually and save the value will remain white. Once a value is set there is no way to allow the field to _not_ have a value.
