Lady Mary´s got brown eyes, neither has Cora or Robert. One cannot inherit brown eyes without any of your parents having it themselves. Either this was only a casting fault, or they have a certain storyline in mind for the Crawleys. Although, it would be quite a scandal, yes, but George would still be the son of Matthew, and heir to Lord Granthams title, hence not a totally ruining storyline.