The Crichton family (who also built the nearby Brunston Castle) had a large castle here on the site of Newhall House certainly as early as 1529, which is said to have covered the whole of the promontory which juts out above the River North Esk.

By 1703 the castle was in a bad condition, and was bought by Sir David Forbes who knocked down most of it, but incorporated the remains of one of the main towers into the front section of the new house’s ground floor. Further additions were made in 1795, and in the mid-19th century.