It did not require a long time, however, for this feeling to react on the higher classes of whites where Negroes settled in large groups.

A strong protest arose from the menace of Negro paupers.

An attempt was made in 1804 to compel free Negroes to maintain those that might become a public charge.[6] In 1813 the mayor, aldermen and citizens of Philadelphia asked that free Negroes be taxed to support their poor.[7] Two Philadelphia representatives in the Pennsylvania Legislature had a committee appointed in 1815 to consider the advisability of preventing the immigration of Negroes.[8] One of the causes then at work there was that the black population had recently increased to four thousand in Philadelphia and more than four thousand others had come into the city since the previous registration.
They were arriving much faster than they could be assimilated.

The State of Pennsylvania had about exterminated slavery by 1840, having only 40 slaves that year and only a few hundred at any time after 1810.

Many of these, of course, had not had time to make their way in life as freedmen.
To show how much the rapid migration to that city aggravated the situation under these circumstances one needs but note the statistics of the increase of the free people of color in that State.