Motherless Brooklyn, by Jonathan Lethem
Upon its publication in 1999, Motherless Brooklyn made waves for its unconventional narrative and another uptick when the author's followup, Fortress of Solitude turned out to be a pretty big deal as far as "things the New York Times really loves" goes. Brooklyn is a fairly stock detective story with one twist: The protagonist, a small-time would-be detective with Tourette's syndrome. The disconnect between the fairly stock (if not labyrinthine) noir story and the main character's condition -- and that he narrates, of course, without the verbal tics that mark his speech -- made for an oddball take on a pretty standard formula. At the time of its release, there wasn't much question that Motherless Brooklyn was a detective story (it even won the Golden Dagger, the big-deal mystery fiction award, that year) but the prominence and undeniable literary fictionality of Lethem's later work has seen its current edition resolutely rebranded "Fiction."
The Master and Margarita, by Mikhail Bulgakov
An old favorite of this humble bookseller, The Master and Margarita is also a classic of Russian literature and a common fixture in college classes. Penguin, which has published it for years and years, comfortably categorizes it as "fiction." Despite this, the novel concerns itself with vampires, witches, giant talking cats andthe Devil himself. Which isn't to say it isn't great -- plus, the first draft was burned, in despair, by the author; the finally published version had 13 percent of its text censored by the Russian government, both of which sound like hallmarks of a great novel to me.
Hard-Boiled Wonderland and the End Of The World, by Haruki Murakami
This 1985 acid trip of a novel -- by the generally excellent Murakami -- owes a lot to its surrealist, absurd premise, concerning a man who uses his brain to encrypt large quantities of data (for hire, of course) and the unfortunate things that happen to him working a case for an elderly scientist. This novel -- Hard-Boiled Wonderland -- alternates chapter-by-chapter with the even more bizarre End Of The World, in which a man moves into a surreal village where he learns how to read dreams by staring at unicorn skulls. Do you follow? While the novel borrows pretty heavily from both cyberpunk and noir traditions, difficulties in classifying it -- and the author's insistence that it didn't quite belong to one of its influential genres over any other -- found it a niche in the US as "surrealist fiction," which, well, it certainly is that.
The Road, by Cormac McCarthy
The Road is probably the biggest line-blur that comes to mind. It lives on the fiction shelf, Oprah loved it enough to put it in her book club, it won the Pulitzer-- so on and so forth. But the novel is about as post-apocalyptic as you could get. With a major motion picture in the theaters as we speak, there's not really any mistaking it for a novel about the end of the world -- it looks and plays out like Mad Max, after all. Maybe Oprah willed it into literary fiction, or maybe the fact that it's really good -- either way, The Road takes the cake as the hardest novel (for me, anyways) to stick on a shelf. Usually, we split the difference, and put it in both sections.