It's all relative.
People often retain a mental image of themselves at around 40, even long after they've passed that age. That self-image tends to remain largely unchanged, so they judge other people's ages relative to it.
As a result, their perception of someone else's age is shaped less by the person's actual age and more by how that person compares to the age they still associate with themselves.
In my opinion, he doesn't look older than his actual age. He looks uncouth and unkempt, which probably creates the impression that he's much older than he really is.