James Russell Miller was born near Frankfort Springs, Pennsylvania, on the banks of the Big Traverse, which according to his biographer, John T. Faris, is a merry little mill stream which drains one of the most beautiful valleys in the southern part of Beaver County. His parents were James Alexander Miller and Eleanor Creswell who were of Irish/Scottish stock.

Miller was the second child of ten, but his older sister died before he was born. James and his sisters attended the district school in Hanover Township, Beaver County, Pennsylvania until, when James was about fourteen, his father moved to a farm near Calcutta, Ohio. The children then went to the district school during the short winters and worked on the farm during summer.

In 1857, James entered Beaver Academy and in 1862 he progressed to Westminster College, Pennsylvania, which he graduated in June 1862. Then in the autumn of that year he entered the theological seminary of the United Presbyterian Church at Allegheny, Pennsylvania.