How can we tell genre fiction from literary fiction? It’s not enough to add some spaceships and call it science fiction. Nor does putting it in a medieval setting automatically make it fantasy.
I think one part of the difference is that genre fiction seems mainly concerned with jobs, and exciting things happening while people are working those jobs: noble, soldier, scientist, private eye.
Literary fiction is less concerned with jobs, and more concerned with life outside of work: families, holidays, dating. Work is implicitly boring, an obstacle to be overcome.
It’s two polar views of the human condition. In one, work is a calling, and the moral questions revolve around what kind of people get called and how they respond to their calling. In the other, work is background. It’s something that may create conflict, but it’s not usually central to the story.
In other words, fiction written in genre circumstances that doesn’t revolve around work as a calling feels literary, even if it’s set in a far-off alien landscape.
Hence Ian McEwan’s novel Solar, which could have been written as genre fiction, following the career of a scientist toward a breakthrough in cheap solar power, but instead is written in a literary style, more concerned with his life outside of his work and what that says about him.
There’s also Ken Follet’s The Pillars of the Earth, which is sold as literary fiction but to me reads like genre: the central plot-line is the construction of the cathedral, and those called to build it. Characters move in and out of the narrative according to their impact on the cathedral’s construction, and there’s a lot of science-fiction-style description of building techniques.