Having spent an inordinate amount of time since the early 1990s listening to all sorts of jazz, including classic jazz, new age, free form, fusion, big band, and various ethnic varieties such as Brazilian etc. what is of course common to all of this is improvisation. But if you ask the question what makes the difference between merely technically interesting improvisation, and improvisation that can be moving and even transcendent it is improvisation that tells a story, that has a narrative line to it and is going somewhere in all those progressions.
So I will quite happily listen to Pat Metheny with joy and anticipation of where the music is going, but I can only listen to Ornette Coleman for so long before I say ‘this is going nowhere’. Some artists, for example, Keith Jarrett have a remarkable innate ability to do instant improv which turns into not merely interesting, but fabulously beautiful progressions and even new tunes. If only he would stop the primal grunts and moans when he is playing. Sadly, we will probably never see him play again in light of the strokes he has had, and the loss of ability in his left hand. But he has so many great albums out there as his legacy.
In regard to ensemble play, I am perfectly happy to listen to excellent solos on the drums that are part of a larger composition involving various instruments, but I doubt I would go to a drums only concert. There is no tune, no melody, no narrative line going somewhere, though a drum solo in a song can indeed help tell the story and play a major part. For me, while I like the classic trios of drums, bass, and either guitar, piano, or some sort wind instrument, I especially like Bill Evans with whoever, nevertheless, for me the the best of all this comes when you have keyboards and guitar ala the Pat Metheny Group, or sax and keyboards along with bass, and drums. Why? Because you’ve got a wider scope of palette to paint from. I have to admit however that the Metheny Mehldau concerts and two albums were wonderfully rich. It’s always better when you have two musical geniuses in the same band who are both classically trained and can improvise as well.
Jazz musicians in any case are real skilled musicians like classical musicians who can play their instruments and understand music theory among other things. I get impatient with those who think rhythmic talking with a back beat qualifies one to be a musician. A beat poet with a back beat maybe, but not a real musician. And that person maybe really good at the poetry, so yes it can be real art form, but it is not the same thing as being a real musician whether, classical, jazz, rock, folk, country etc. And I would say the same thing about those who can manipulate technology to make music, but can’t actually play real musical instruments or sing etc. Don’t get me wrong, some of that techno stuff can be fun and even musical in spurts but if supporting real musicians and real music, not artificial stuff is important, and it is especially in jazz, then let’s call things what they are.
I realize I’m old fashioned in these things in some respects, but having played and sung classical, jazz, rock, country, Gospel music for a long time, you can spot a real musician a mile off when you’ve had the musical experiences I have had and seen hundreds and maybe thousands of concerts over my 70 years.