The Van Cortlandts, a merchantile family prominent in New York affairs, established a grain plantation and grist mill on the property. The house was used during the Revolutionary War by Rochambeau, Lafayette, and Washington.
After 140 years of occupancy by the Van Cortlandt family and their slaves, in 1889 the property was sold to the City of New York and made a public parkland.