Ballyragget Castle, also known as Butler Castle, is a ruined square tower house. It has defensive wall with corner towers. It is named after the Norman, Richard Le Ragget, who owned the land in the 13th century.
Ballyragget Castle was built in 1495 by the Mountgarret family. In 1600 it was garrisoned by Sir George Carew against the Mountgarrets who were in opposition to the crown. The castle was once a residence of Lady Margaret Fitzgerald, the Countess of Ormonde.