Seton Castle, or Seton Palace as it was also known, was built in the 15th century by George Seton, the 4th Lord Seton, although its origins may go back to the 12th century.
By the late 16th century it consisted of an L-plan house with a tall square tower projecting from its centre. A range of buildings were enclosed by a courtyard wall, at the corners of which were tall round towers. The entrance consisted of a small doorway flanked by two massive round towers.
When George Seton, the 5th Earl of Winton supported the Jacobites in 1715 he forfeited his estates, including Seton Castle and Winton Castle. The now ruined Seton Castle was sold to the York Buildings Company, who sold it on to Lt Col Alexander Mackenzie of the 21st Dragoons.
Lt Col Mackenzie commissioned Robert Adam to design a new house for him, and in 1789 Seton Castle was demolished to make way for the new building, which was completed the following year.
Some time after the death of Lt Col Mackenzie in 1796 the house passed into the hands of the Earl of Wemyss.
In the mid-19th century it was apparently possible to still see the remains of the castle’s prison to the west of the later house, with a chamber measuring 2.7 metres by 2.1 metres with walls some 1.4 metres thick.
Seton Castle is now a private residence, but several heraldic panels from Seton Castle are now on display in the grounds of the neighbouring Seton Collegiate Church (managed by Historic Scotland).