If you feel as though you aren't getting much fat at all, then adding in full fat dairy won't hurt.
If you eat cheese, avocado, oily fish like salmon, meat with some fat etc then sticking with lower fat options sounds fine to me.
Full fat milk is approx 4%. Skim is approx 2% fat. There's only a 2% difference!
I would rather have less full fat milk than the more processed, sugar laden alternative.
The difference is neglible between ordering a full fat cappuccino rather than a skinny latte.
There's also the difference between fats. Not all fats are the same or 'good' for you. Monounsaturated fats are 'better' for you than saturated fats. Don't even get me started on trans fats. You can't substitute one for another (in foods) and expect them to be the same in the body.