Patrick seems to have been a long lived Earl. He had to be of age in 1825, otherwise his father would have made the decision, to grant a building to the town, placing his birth in 1807. We know he fought in 1853 - 1856 in the Crimean War and that he married in 1860 and was alive in 1890 (Cora mentions him making her sign over her fortune to the estate) and died sometime after 1895 - as Edith mentions that he left the grandchildren a trust fund, which would presumably include Sybil.
Born in 1807, he was 35 years older than Violet, was unmarried (or, widowed with no children), at age 53 and died aged at least 89.
This means that, at the time of his birth, his father and grandfather had not become Earls yet as the 2nd Earl was still alive, and presumably young (as his mother ordered him home and ordering home an adult presumably wouldn't work).
This implies that his father and grandfather were Earls for no more than a period of about 85 years between them: 1789 - when the second earl was LAST known to be alive - and 1874 when Violet mentions her husband specifically as "Lord Grantham".