The castle is situated on the Laich of Moray, a fertile plain that was once the swampy foreshore of Spynie Loch. This was originally a more defensive position than it appears today, long after the loch was drained.
The motte is a huge man-made mound, with steep sides and a wide ditch separating it from the bailey. The whole site is enclosed by a water-filled ditch, which is more a mark of its boundary than it is a serious defensive measure. Duffus Castle was built by a Flemish man named Freskin, who came to Scotland in the first half of the 1100s. After an uprising by the ‘men of Moray’ against David I in 1130, the king sent Freskin north as a representative of royal authority.
He was given the estate of Duffus, and here he built an earthwork-and-timber castle. Freskin’s son William adopted the title of ‘de Moravia’ – of Moray. By 1200, the family had become the most influential noble family in northern Scotland, giving rise to the earls of Sutherland and Clan Murray. In about 1270, the castle passed to Sir Reginald Cheyne the Elder, Lord of Inverugie. He probably built the square stone keep on top of the motte, and the curtain wall encircling the bailey. In 1305, the invading King Edward I of England gave him a grant of 200 oaks from the royal forests of Darnaway and Longmorn, which were probably used for the castle’s floors and roofs.