If per-face color is "Empty", then this face does not have a custom color