He always went by the name "Dudley", and all official records that he was required to enter his name was always "J. Dudley Roberts", never James or James Dudley. He was the oldest of 7 children, and in his early years, he worked on the family farm in Jena, grew truck crops and sold the produce and vegetables from a wagon which he drove around town drawn by a horse.

During the depression years, he was working in the CCC (Civilian Conservation Corps) and he enjoyed going to the annual reunions of that group whenever he could.