Why is my scrush so ineffective

I find that most games where i decide to scrush i get walled out or they quickly spear up and guard their lines. I get much more joy over m@a - archers or drush - anything than i do with scouts. 5 scouts total = 400 food which puts my castle age time at 20 minutes. Also, scouts kind of obselete themselves in castle age. I dont tend to light cav as knights are a bigger threat by then anyway. What am i doing wrong? Is there a very small early feudal window that i have to nail for them to be effective?