Is it really that game breaking if patrolling units ignore farms?

It really annoys me to see the bulk of my army patrol into enemies' farm and stayed there for 10 minutes before they moved on to other buildings, coz you know, farms are such a big threat that needs to be terminated immediately.

I just wonder if there is anyone who thinks this would jeopardize the game balance.