How to properly handle a forward + castle drop?

I was Teutons against the byzantines and got steamrolled by a strong forward and gg’d on what would soon be 4 castles right in my face. I had knights on the field but I lost them all in an unfavourable fight against pikes and xbows when I tried taking one of his castles down with some rams (to no avail). I tried mixing in skirms but it was too late and he basically had the whole map controlled so I gg’d out What should I have done other than trying to get to imp and start trebbing the castles? Also on a side Note, when you try to mix some trash in, like i did with skirms to counter his archers, should you always upgrade them fully? Or rather spend those res on numbers, or upgrades for your main gold unit?