He was broken-hearted at seeming to be wanting in his sense of honour and duty.

People, he says, accused him of putting the money which had been paid for the tomb out at usury, "living meanwhile at Florence and amusing himself." It also hurt him deeply to be distracted from the cherished project of his early manhood in order to superintend works for which he had no enthusiasm, and which lay outside his sphere of operation.
It may, indeed, be said that during these years Michelangelo lived in a perpetual state of uneasiness and anxiety about the tomb of Julius.
As far back as 1518 the Cardinal Leonardo Grosso, Bishop of Agen, and one of Julius's executors, found it necessary to hearten him with frequent letters of encouragement.

In one of these, after commending his zeal in extracting marbles and carrying on the monument, the Cardinal proceeds: "Be then of good courage, and do not yield to any perturbations of the spirit, for we put more faith in your smallest word than if all the world should say the contrary.

We know your loyalty, and believe you to be wholly devoted to our person; and if there shall be need of aught which we can supply, we are willing, as we have told you on other occasions, to do so; rest then in all security of mind, because we love you from the heart, and desire to do all that may be agreeable to you." This good friend was dead at the time we have now reached, and the violent Duke Francesco Maria della Rovere acted as the principal heir of Pope Julius.
In a passion of disgust he refused to draw his pension, and abandoned the house at S.Lorenzo.This must have happened in March 1524, for his friend Leonardo writes to him from Rome upon the 24th: "I am also told that you have declined your pension, which seems to me mere madness, and that you have thrown the house up, and do not work.
Friend and gossip, let me tell you that you have plenty of enemies, who speak their worst; also that the Pope and Pucci and Jacopo Salviati are your friends, and have plighted their troth to you.

It is unworthy of you to break your word to them, especially in an affair of honour.