Is number of villagers or advance time more important?

I can hit advance times well but my villager count is usually lower than suggested by build order templates.