Deleting Game Folder + Packs I don't want

Is it really necessary to delete the 'Game' folder every time I install a new pack since it's replacing the files? Or does it leave some files behind that I shouldn't have? I've forgotten to do this several times but my game works fine. I'm worried it might mess up something in the future?

There are a couple packs I don't want to have in my game, like the Moschino pack. Do I have to install it in order for everything to work smoothly? And if I decide in the future I want to uninstall a pack, what should I do (if it won't mess anything up)? Thanks!