In 1476, during the Duke of Burgundy's first campaign against the Swiss, the more or less active participation of the Duke of Nemours with the king's enemies appeared to Louis so grave, that he gave orders to his son-in-law, Peter of Bourbon, Sire de Beaujeu, to go and besiege him in his castle of Carlat, in Auvergne.

The Duke of Nemours was taken prisoner there and carried off to Vienne, in Dauphiny, where the king then happened to be.

In spite of the prisoner's entreaties, Louis absolutely refused to see him, and had him confined in the tower of Pierre-Encise.

The Duke of Nemours was so disquieted at his position and the king's wrath, that his wife, Louise of Anjou, who was in her confinement at Carlat, had a fit of terror and died there; and he himself, shut up at Pierre-Encise, in a dark and damp dungeon, found his hair turn white in a few days.

He was not mistaken about the gravity of the danger.