Castletown itself developed around the 12th century castle which originally stood at this location. All that remains today is the mound (motte) which the original castle stood on which is still visible in the garden of Castletown House. The original Norman borough of Castletown would likely have stretched from the motte at Castletown House to the site of the present church just down the road.
The Archaeological Inventory Report on Castletown states that "Castletown was the site of one of the most important Anglo-Norman boroughs in Laois." The Anglo-Normans of the 12th century are only known to have established three boroughs in Co. Laois and Castletown was one of these. The earlier Irish name for Castletown, Baile chaisleain Ua mBuidhe, would be consistent with the name which the Norman's gave to their castle which was Castle Oboy. This castle would have been the manorial home of Robert de Bigarz who was granted this area before 1176 by Strongbow. The castle was built in 1182 for Bigarz by Hugh de Lacy and passed through several hands in the subsequent years. In the 1300's the borough at Castle Oboy was first referred to as Castleton. The town would have been quite prosperous in the 12th and 13th centuries but then declined rapidly with the decline of the Norman fortunes after the 14th century.