Fages and Serra did not work well together, and, at the time of the founding of San Luis Obispo, relations between them were strained almost to breaking.

Serra undoubtedly had just cause for complaint.

The enthusiastic, impulsive missionary, desirous of furthering his important religious work, believed himself to be restrained by a cold-blooded, official-minded soldier, to whom routine was more important than the salvation of the Indians.

Serra complained that Fages opened his letters and those of his fellow missionaries; that he supported his soldiers when their evil conduct rendered the work of the missionaries unavailing; that he interfered with the management of the stations and the punishment of neophytes, and devoted to his own uses the property and facilities of the Missions.
In the main, this complaint received attention from the Junta in Mexico.

Fages was ultimately removed, and Rivera appointed governor in his place.