How do you make garrisoned villagers return to a previous job without ringing the bell?

I just saw a game with theviper where he quickly boxes his farmers and garrisons into his TC for a few shots on a scout and then immediately has them return to their farms. When I try that my vills go to my rally point