I'm not sure I understand where the fanservice line gets drawn musically. My mom doesn't like hardcore metal, but is that because it's self-referential fanservice music or because she just doesn't like it? I mean, couldn't you chalk up any taste preferences to fanservice?
"Fanservice" doesn't just mean music with a cult following that a majority of people happen to find alienating -- that would, after all, condemn virtually everything I have ever done musically as fanservice. (Although maybe that's actually a fair assessment, I dunno.) Similarly, "fanservice" does not cover all music that is deliberately constructed to alienate people, nor does it cover all music that might be a bit of a tough slog for the uninitiated.
Fanservice is more specific than that -- it's a marker or signifier that serves no legitimate aesthetic purpose, but is there to stroke those in the in-crowd while simultaneously alienating even the most sophisticated and open-minded newbies.
Of course, the devil is in the details -- what counts as a "legitimate aesthetic purpose"? -- etc, etc. There is also the inconvenient truth that fanservice works -- it's fun to be part of a clique, even if your clique is widely regarded as uncool. (Sometimes especially if.) It's fun to spot obscure in-jokes or references that go over everyone else's head. It's fun to feel like you and your friends are musically knowledgeable/sophisticated/progressive/hardcore and everyone else is a bunch of pikers. Basically, it's fun to be pandered to.
So here are some examples of what I'd consider fairly unambiguous musical fanservice:
• ensemble orchestrations of classic jazz solos (Supersax, Lambert, Hendricks & Ross, etc). The only example I can think of where this actually works is Hal Overton's chart on "Little Rootie Tootie" from the Thelonious Monk Orchestra at Town Hall record.
• all of the hoary rituals surrounding classical concertgoing -- the no-clapping-between-movements rule, the taboo against speaking to the audience, the ridiculous tuxedos, various and sundry other bits of formalized pandering.
• the spiteful parody of Shostakovich 7 in the fourth movement of Bartók's Concerto for Orchestra, which is never actually funny. (Okay, almost never. But it's still a nasty bit of fanservice that seriously detracts from my enjoyment of what is otherwise one of my favorite Bartók works.)
• Milton Babbitt. "The Composer As Specialist" (aka "Who Cares If You Listen") is essentially one long defense of fanservice.
[And here is the official fanservice component of this post: add your own examples in comments!]
Keep in mind, some of the above I have actually enjoyed at one time or another -- I am certainly not immune to fanservice. And if the thing as a whole is good, it's easy to forgive the occasional burst of fanservice, even if it detracts or distracts. It's only when the entire work/subgenre/scene is built around creating and rewarding fanservice that we start to have a problem.
A commenter going by the name of medrawt said something else that I think is very pertinent:
Somewhere online I read someone say that essentially the "ideal" reader of Iron Man or Spiderman has ready every issue of the comic, has read all crossover issues with different titles, recalls all of them, and has the additional capacity to spontaneously forget those issues whose continuity would render the current issue unintelligible, until such time as some future writer chose to reference them in some way.
Here's one failsafe way you can tell if you, as a artist, are indulging in fanservice -- if you ever find yourself thinking anything remotely like this:
The ideal listener of my music has heard and analyzed everything I've ever created, is intimately familiar with all of the works that have ever influenced me and everyone I've ever collaborated with, has read all of the written statements I've made about my music, recalls all of this, and has the additional capacity to spontaneously forget those works and/or statements which have no bearing on my current output, until such time as I chose to revisit that creative period in some way."
1. Okay, if you clicked that link, I know you want to listen. I have been "enjoying" The Most Unwanted Song for many years, long before it became an internet phenomenon -- Society co-conspirator actually owns the CD, and likes to break it out at parties.