Ayton Castle is a largely 19th century mansion however it was built on the site of a castle which dated back to the 11th century.

The original castle may have been built by the Norman lord Gilbert De Vescie, however it was owned from the 11th century by the Aiton or Ayton family. It later passed by marriage to the Home family.

In 1448 the peel tower of Ayton was destroyed by the English and then after rebuilding was captured by the Earl of Surrey following a siege in 1498. Some sources conflate the “pile of Ayton” (i.e. Peel of Ayton) “taken with the barmkyn with divers villages and steads thereabouts burnt” in 1542 with Peelwalls, however this may actually refer to Ayton Castle. As a strategically-important Border stronghold it was held by the English between 1547 and 1550.

The Homes were forfeited after the first Jacobite Rising in 1715 and by 1765 the castle had passed to the Fordyce family.

In 1834, by which time it had been extended and remodelled as a classical mansion, it burned down and between 1845 and 1851 it was replaced with a mansion designed by James Gillespie Graham. Further additions were made by David Bryce in 1860 and 1861.