The earliest version of Crawford Castle consisted of a motte and bailey, with a surrounding ditch. On the remains of the motte stand the ruins of a square enclosure surrounded by a curtain wall. Researchers believe there may have been round towers at each corner of the enclosure dating back to the 16th or early 17th centuries.

A range of buildings on the southwest were also constructed around the 16th or 17th century. This tower-like range was three stories tall, with an attic and a vaulted basement. A second range was added in the 17th century to the southeast. This addition created more spacious accommodation for the Clan Crawford members.

Much of the currently visible ruins likely date from the 17th-century rebuilding of the castle by the Marquess of Douglas. Stone from the old castle was likely robbed to construct the nearby Castle Crawford.