At one time, First Presbyterian Church at Caldwell owned most of the center of Caldwell, when it was known as Horseneck. The church sold the property to build the present church. The name of the community was changed to Caldwell after James Caldwell, the fighting parson, who served as deputy general in George Washington's army and helped establish the church.
The second pastor was Richard Cleveland who named his first son Steven Grover after the first pastor. Grover Cleveland became the 22nd and 24th president of the United States. The Baby Ruth candy bar was later named after Ruth Cleveland, the first baby born in the White House.
The First Presbyterian Church at Caldwell, with its soaring steeple, is still today a focal point in Caldwell, much like it was more than 200 years ago.