Once or twice a week he comes in here, and we talk.

That is his chair--the one you are sitting in.
Once or twice a week I go up and sit in his chair and talk.

In all the years I have known him I have only seen him troubled once or twice.

Then I asked him the reason, and he told me.

To-day I heard you speak about some money you wanted, and then I saw that something had gone wrong.
After I left he came downstairs and passed my window and did not look in.