Why the cr*p is it so hard to repair buildings? Or am I doing something wrong?

A lot of the time if I try and repair a palisade to prevent the enemy breaking it down, I can't get the hammer icon no matter where I hover my mouse, just a sword to attack the enemy with.

Is there a fix for this?