Part of what has come to be known as the frontier defenses between England and Scotland, Cessford Castle is a 15th-century castle that was once the stronghold home of the Ker family.

It is thought that Andrew Ker began building this castle around 1450. The Ker family spent much of the 1500s in a bitter feud with the Scotts of Buccleuch, a fight that began after a man working for the Kers was killed by a Scott.  In an attempt to bring peace to the families, marriage contracts were set up, but largely unsuccessful.

Cessford Castle is an L-shaped tower house, it was built with four-meter-thick defensive walls. Inside, there was living space for not only the family and their servants, but also for a garrison of up to 60 soldiers. The castle was also defended by an earth bank and ditch running around the outside. There is also the remains of forework which would have added another layer of protection.

The castle was frequently attacked by the English, with one spectacular siege in 1523 by the Earl of Surrey. The Earl's men climbed the outer wall of the castle, covered by cannons and archers, but they could still not gain entry. Following this failed attempt two smaller cannons were used to open up a hole in the castle wall through a blocked-up window and barrels of gunpowder placed inside.  The Scots blockaded inside the castle set fire to the gunpowder before the English could blow it up, and successfully subverted that attack.

The castle continued to act as the Ker family home until 1607, when Sir Robert Ker moved to a different property close to Melrose. It was his great-grandson that went on to have nearby Floors Castle built after the family took the titles of Earl and then Duke of Roxburghe.

The ruins are said to have held a prison pit deep within the castle which could only be entered through a hatchway in the chamber above, with a single, small flue for ventilation.