Issue: farmers garrisoned in TC don't go back to their farms after being sent "back to work"

I garrison farmers in TC (due to enemy around) and when I click "back to work", they walk back to their farms but just stand there. I then have to put them back to work manually. Why don't they go back to work automatically?

(I play on Voobly)