Like you say, how/why a thing is genetically modified matters. Changing something, like rapeseed for canola oil, through selective breeding and evolution is one thing. But when people talk about GMOs, that's not what they're talking about.
Then you have "BT corn" - supposedly it's safe for humans because it's just gene splicing in a naturally occurring soil bacteria, Bacillus thuringiensis, that kills off certain bugs but is selective in what it kills - but is FDA approved to be safe for humans. Then again, the FDA has approved HOW many drugs that they then end up going back and black-box warning-labeling 5-10 years later? Other GMOs are "Roundup Ready" crops - meaning they are resistant to the herbicide Roundup so commercial farmers can spray on more Roundup without worrying about killing their crops.
The problem is that just because something doesn't kill you doesn't mean it isn't inflammatory to your gastrointestinal track.
In general, with any food, the less processed you can get it the better. I tend to match my oil to whatever I'm doing with it - high heat cooking I go for something with a higher smoke point. Usually coconut oil. I don't personally have issues with saturated fats from non-animal sources (and even some animal sources, really, if they aren't CAFO sourced). If I'm making mayo or salad dressing I use olive oil.