In Henry Eyring's General Conference talk in October 2017 he mocked those leaders who declared Joseph Smith a "Fallen Prophet". Before the end of 1838 all the following significant leaders had left: (Eyring's Talk)
Oliver Cowdery, Martin Harris, David, Peter, John Whitmer, Hiram Page, Thomas Marsh, William McLellin, Luke Johnson, their families and many more.
Here is a list of some of the "insignificant reasons" that they may have left.
Real Reasons - Spilled milk, lazy, wanting to sin, misspelled names, offended.