After he was confronted about Peter Gordon, he just left shortly after. We never really find out if he was really Peter trying to pass himself off as Patrick Crawley or actually Patrick Crawley and too distraught that nobody would believe his story. We are only told that when he left he said it was "too much, and he couldn't go thru with it".