Originally part of the Lord Fairfax patent in the early 1700's, this land was deeded to Thomas Fairfax (the 9th Lord of Fairfax) in 1790. It had served as a family hunting lodge. Refusing to take the titular name, this Fairfax wished to be called "Mr. Tommy Fairfax". He educated his slaves (rare) and set them free (rarer) though they reportdly came back after a few years asking to be taken back in. His scientific achievements include inventing methods to channel lightning strikes and worked on galvanizing steel.
The second son of this Fairfax, Henry was killed in the Mexican American war and the widow sold the land to a northern immigrant, James Sherman a recent arrival from Cayuga County, NY. Sherman's son, Franklin served with the 10th Michigan Cavalry in the Civil War and saw action- predominantly in the western campaigns in Tennessee. Franklin married a woman who taught at the Freedman's school next to Ash Grove.
The Sherman family owned the land from the 1850's until 1997 when they conveyed it to Fairfax County. About 1960, the main house burned down and what you see is a re-creation similar to the original style. Major parts of the land was sold off piece by piece- some for the Toll Road exchange, some for the local hotel.
Note: while this can be a quick P&G if someone stays by the car, there is not really good parking nearby. I have listed the place where I parked as a waypoint.