In Season One Violet says that Crawley House 'always seemed so dark when my mother-in-law lived here.'
If this woman, (Robert and Rosamond's grandmother) lived at Crawley House then who lived at the Dower House? If it had have been her mother-in-law (Robert's Great-Grandmother) then that would make sense but it seems unlikely that this woman would outlive her son so that her grandson (Robert's father) inherited.
Would it be possible that when they were first married Robert and Cora lived in the Dower House? I can definitely see a lot of friction between Cora and Violet when Robert's father died and Violet moved to the smaller house whilst Cora became the mistress of the Abbey.