There was a tower house at Milton which belonged to the Blair family.
Milton was the main residence of the Blair family in the 16th century.
In the 18th century a new house was built which incorporated part of the old tower. Milton House was subsequently remodelled in 1885.