Southside Castle is an L-plan tower house which is said to have been built by a Patrick Ellis between 1640 and 1644. However it is likely that Ellis extended and remodelled an earlier castle, as its appearance is older than the 17th century.

By 1850 it was semi-ruinous, but it has now been restored although with many alterations. It has been reduced from its original height of four storeys to three, the vaulted ceilings have been removed from the wing (that in the basement of the main block is still intact), and a two-storey extension built in the re-entrant angle of the two sides.