If you had said that on this point you disagreed with him, his manner probably would have implied that he thought you a bit of an ass.
When a new man arrives in Thrums, the women come to their doors to see whether he is good-looking.

They said No of Tommy when he came back, but it had been an emphatic Yes for Dr.Gemmell.He was tall and very slight, and at twenty-seven, as at twenty-one, despite the growth of a heavy moustache, there was a boyishness about his appearance, which is, I think, what women love in a man more than anything else.

They are drawn to him by it, and they love him out of pity when it goes.

I suppose it brings back to them some early, beautiful stage in the world's history when men and women played together without fear.
Perhaps it lay in his smile, which was so winning that wrinkled old dames spoke of it, who had never met the word before, smiles being little known in Thrums, where in a workaday world we find it sufficient either to laugh or to look thrawn.

His dark curly hair was what Grizel was most suspicious of; he must be vain of that, she thought, until she discovered that he was quite sensitive to its being mentioned, having ever detested his curls as an eyesore, and in his boyhood clipped them savagely to the roots.