The legend says that the castle was named after the giant Isoré, a character out a story by Chrétien de Troyes, a French writer of the 12th century. Isoré was a Moorish king who attacked pilgrims travelling to Compostella, robbing them of their possessions and their lives. A hermit, called William of Orange, challenged him and cut his head off, freeing again the way to Compostella.

More probably the name comes from the Latin Isuriacum or the property of Isurius, a Gallo-Roman nobleman.

The first known owner is Jean d’Armagnac in 1581. The castle, still a manor at that time because it possessed no towers, consisted solely of the east wing and the hexagonal staircase. Jean d’Armagnac was the first chamberlain of the French king, as was his son after him. The family coat of arms with the red lion on a silver field can be seen above the entrance of the staircase tower. During the 17th century important changes and additions took place. The North wing, the barn and the farm buildings were added.

After 1714 the Armagnacs sold the property, and there were a series of short term owners. In 1914 the castle was requisitioned by the French army and used as a sanatorium for officers with battle shock. That this was not the best usage of the building was shown by the fact that the last noble owner, Jeanne Vatar des Aubiers, sold the domain to a farmer’s family.

In 1983 the castle was bought by the council of Avoine to transform it into a home for the elderly. This was considered not realistic and the building was again put on sale on the private market until it was acquired in 2009 by the actual owners.