AoE2:DE Unplayable lagg further into games

I'm experiencing really bad lag later on in games. My benchmarkscores are 1400+.I'm running an I7 10700K, MSI Radeon RX 6700 XT and 32GB of ram. I really doubt my hardware is the issue.

The grame runs smooth and perfect at the start, but near the end by the time i get to castle age/imp, I start having constant 3-5second long freezes and huge imput delay. I'll select a bunch of troops and order them to attack, and only after like 10 seconds will they start moving, and at a really fast speed when the game's trying to catch up.My friend who i'm playing with hardly has any trouble.

Is it just my internet connection that's failing? Any known issues? I've googled around but haven't found anything helpful or more recent than a year.

EDIT: This is in multiplayer games with the normal DE AI (moderate/hard, we're not that good xD)