I think that the bad relation was because in the end both Mary and Edith were women who in reality had to compete to get a man in order to get married, that was their "job" (have a proper husband with money and tittle) and the show was very explicit in let us know that Mary was more prettier and more accomplished in the small talk, the qualities that a woman of her time required; Edith´s habilities were somewhere else (as Sybil pointed out during the war). Sybil in the other hand only "came out" in 1914 the same year that the war started, so she didnt participe in that competition, but it was very clear if the war wouldnt change the way that things worked Mary and Edith would have seeing her as a competition also, remember the face that Mary have when Matthew saved Sybil of the politic rally and she inmediatily jumped towards Matthew.
Its a sad thing, but that is how thing worked for women of the aristocracy back then, even Mary complained about it saying that she was basically in a golden cage waiting for marriage and in the end she was jelaous of Matthew because he had a life with purpose besides marriage. So basically i think that Mary ane Edith competed for the attention of men during all their teenager years and Mary is the "heir" of Violet, she can be gracious but very cunning if the moment requires it.
The ww1 changed all of that, women suddenly have more freedom and option besides a good marriage, this produced dramatic changes in Sybil but also Edith. I think that Edith following Strallan and trying to marry him was her last attempt to back to the old days. And we must remember something, in any moment, besides taking care of Mattthew Mary involved herself with the war; for her it was only a little annoyance to her personal life. For that reason, all those changes didnt affected so much. So, after the war, we have that Mary is still behaving herself as a woman with pre war values, and in the other hand Edith after being jilted triied to carve her destiny somewhere else, in conclussion there wasnt a competition anymore in Edith´s side, she even asked Mary that they could get along better but Mary rejected that because Mary was still set up with the pre war values. If we see in the show no longer Edith makes anything to Mary, for every 5 insults of Mary towards Edith, Edith made 1 only to be beaten again by Mary´s shapest tongue.
After Matthew death, Mary suddenly realizes that she must work as anyone else, that she is no longer to be the next countess something very important for her. But even she didnt mature, she continued to see Edith as a competitor, not as her sister, who in the last years didnt care about a propper marriage or nothing else to compete, even Edith knew that her parents didnt love her in the same way that they loved Mary. Also please notice how Mary behaves with other women who can be a threat, ex. Mabel Lane Fox. Then if we see season 6 Mary is having trouble with Edith succes with her magazine in London, she puts a face when Robert praised Edith for the editing thing. Then she mocked about Edith going out with Bertie, again, Edith didnt care that he was a estate agent of Brancaster or his prospects. But when Bertie inherits Mary instantly became jelaous, because in her mind its imposible that a woman like Edith could out rank her, because she thinks that its illogic that a marquess would preffer someone like Edith who doesnt have her beauty or her graces, and off course sabotage everything in order to restore the natural order of things, Edith always down to her.
We must remember that she was surprised about how well Edith "knew" her intentions that she finally accepted how low went this time, and also the words of Violet and Robert. And in some ways she matured.
So my bet, is that their bad relation is result of various things: in one hand how bad Cora made her job raising them, the times that their lived, and the lack of maturity in Mary´s part.