General Grant, as early as 1862, was making further use of them as fatigue men in the department of the surgeon-general, the quartermaster and the commissary.

He believed then that such Negroes as did well in these more humble positions should be made citizens and soldiers.[24] As a matter of fact out of this very suggestion came the policy of arming the Negroes, the first regiment of whom was recruited under orders issued by General Hunter at Port Royal, South Carolina in 1862.

As the arming of the slave to participate in this war did not generally please the white people who considered the struggle a war between civilized groups, this policy could not offer general relief to the congested contraband camps.[25] A better system of handling the fugitives was finally worked out, however, with a general superintendent at the head of each department, supported by a number of competent assistants.

More explicit instructions were given as to the manner of dealing with the situation.

It was to be the duty of the superintendent of contrabands, says the order, to organize them into working parties in saving the cotton, as pioneers on railroads and steamboats, and in any way where their services could be made available.
Where labor was performed for private individuals they were charged in accordance with the orders of the commander of the department.