Elie House is a late 17th century mansion which was built on the site of an earlier castle and extended and remodelled in the 18th, 19th and 20th centuries.

Around 1690 Sir William Anstruther of Anstruther bought the lands of Ardross, together with those of Elie, Bruntshiels, Muircambus and Carmurie, and in 1694 he was granted a charter by Queen Anne “of the baronies of Anstruther and Ardross, with many other lands”. He commissioned the grand new mansion of Elie House to be built in 1697 and the Anstruthers moved there, leaving their seat of Anstruther Place to be tenanted by old servants of the family.