He was the fourth of eleven children by the same mother, who is described in the pedigree of the family given in the Herald's visitation of the county of Stafford in the year 1663, signed by Dud Dudley himself, as "Elizabeth, daughter of William Tomlinson of Dudley, concubine of Edward Lord Dudley." Dud's eldest brother is described in the same pedigree as Robert Dudley, Squire, of Netherton Hall; and as his sisters mostly married well, several of them county gentlemen, it is obvious that the family, notwithstanding that the children were born out of wedlock, held a good position in their neighbourhood, and were regarded with respect.

Lord Dudley, though married and having legitimate heirs at the time, seems to have attended to the up-bringing of his natural children; educating them carefully, and afterwards employing them in confidential offices connected with the management of his extensive property.

Dud describes himself as taking great delight, when a youth, in his father's iron-works near Dudley, where he obtained considerable knowledge of the various processes of the manufacture.
The town of Dudley was already a centre of the iron manufacture, though chiefly of small wares, such as nails, horse-shoes, keys, locks, and common agricultural tools; and it was estimated that there were about 20,000 smiths and workers in iron of various kinds living within a circuit of ten miles of Dudley Castle.

But, as in the southern counties, the production of iron had suffered great diminution from the want of fuel in the district, though formerly a mighty woodland country; and many important branches of the local trade were brought almost to a stand-still.

Yet there was an extraordinary abundance of coal to be met with in the neighbourhood--coal in some places lying in seams ten feet thick--ironstone four feet thick immediately under the coal, with limestone conveniently adjacent to both.