Follow-up to a drush

I'd like some help with how to follow up a drush. Like I understand the point is not to do heaps of damage but more put your opponent on the defensive to free you up to play a preferred strategy. I played a Lithuanian game recently where I forced an opponent to quick wall off with the 3min drush (2 militia) and denied his berries, but he still beat me to castle because I went for scouts after (he went archers into xbow) looking at the eco after we were actually level on vill count but the scouts really ate up my food and I couldn't get castle. Would I have been better to just mirror him and go ranged units? He was playing mongols on Arabia.