So you're not making statutory deductions and paying employer's NI? Do you have employee liability insurance?
The OP does not have to unless he is a company employing the person
The Handyman is responsible for his own statutory payments
Simply not true.
If the handyman is classed as an employee, which working for the one 'client' he almost certainly is, then the employer must register as such with HMRC and make the appropriate deductions.
Being a company has nothing to do with it. A sole trader or even a private individual not in business can be an employer.