The stone mansion, erected in the early eighteenth century by John Van Cortlandt (eldest son of the manor's first lord), was over 150 years old at the time of Lossing's visit. The original family name was Stevens or Stevensen; "Van Cortlandt" derived from "van (from) Courland," referring to the Duchy of Courland in Russia. The family emigrated to Holland, then to America, settling in New Amsterdam.

In 1697, Stephen Van Cortlandt purchased the large Hudson estate. The manor featured thick stone walls with loopholes for defense against Indigenous attack. A pleasant lawn fronted the house, commanding extensive views of Croton Bay and the Hudson beyond.

Historic associations included hosting preacher Whitefield, Governor Tryon and his secretary Edmund Fanning (who attempted bribery in 1774 to secure General Van Cortlandt's crown allegiance — which he refused), George Washington as a frequent guest, and Benjamin Franklin. The old fort or castle of Kitch-a-wan, reportedly one of the most ancient Indian fortresses south of the Highlands, stood near the cemetery at Croton Point's neck. Sachem Croton built this fort for hunting and war assemblies.