In the 2013 Christmas Special, Tom says that each of the shields on the wall are the coat of arms of the Earls of Grantham and their wives.
This means there are 5 shields on the left, three across the top, five on the right and three across the bottom just off screen to make up the fourth balcony. This, in total, makes sixteen Earls of Grantham (presuming my maths are correct: 5 + 5 + 3 + 3 = 16).
Presumably, this means that there have been 16 Earls since 1772 as it's been confirmed that Downton came with the first Earl when he was made Earl.
This, of course, means that Robert is the SIXTEENTH Earl of Grantham, as a shield would presumably have been added when he and Cora got married; you know, tradition and all that.
If so, this means Robert's father was the 15th Earl.