Anything less than complete Spiritual awakening is a dangerous thing. Ignorance is indeed bliss. Once a person has any pretensions about relationship with God as religious or even spiritual seeker, ego is a natural compatriot. The more one knows, the more ego one acquires. Rigidity is a natural consequence. Ego dogs one until the point of awakening, unless one is razor sharp in his/her intellect, or has a Guru who keeps making sure that ego is pricked every time it gets bloated, or one is in complete devotion (where nothing really matters).
As Sadhguru Jaggi often says, that Spirituality in uncommitted environment is a dangerous thing, therefore spiritual processes were practised in very special way and under strict supervision. Spiritual processes in India were experimented with and nothing was out of bounds. Guru, one who has mastered the processes, played an important part in such a journey. It is important for a Guru to oversee the process because one who is on the journey is very prone to having inflated ego.
Since this wasn’t offered on public square, but only in controlled ways and to people desirous of going through the path, it was called a “Secret path”. Secret, not because it shouldn’t be told to anyone, but because it cannot be publicly disseminated without any regard to proper atmosphere.
In that sense, Spiritual path has a certain rigor.
These Spiritual Masters also acted as Guides and Priests to the society and kingdoms of that age. Kings were known to have them in special esteem for proper advise and blessing. One of the first thing to happen on Spiritual awakening (Enlightenment) is Clarity of Vision. That clarity of vision was important to the decision making for the kings. That is what they were after mostly.
If proper checks and balances aren’t kept in Spiritual processes and environments, things can deteriorate pretty quickly simply because one is no longer passing on of the experience of the Truth. It, instead, becomes a passing on of knowledge. Knowledge is at best an estimation of Truth… never the Truth itself.
Spiritual Processes require a certain rigor to make them useful and productive, but when they deteriorate into passing of knowledge, the rigor transforms into rigidity.
Passing on the experience of Truth to a select few with complete rigor is a worthy endeavor. But passing on knowledge to a select few with all sorts of rigidity in the system is a caricature.
Such a thing can only happen when a Spiritual Master is not able to prepare Realized and Spiritually awakened beings in his own lifetime and leaves his students “half-finished”. No matter where you are in your SPiritual journey.. anything less than full awakening is accompanied with only knowledge. And turning knowledge into awakening is not easy for knowledge is directly proportional to ego.
The journey from Spiritually advanced society that existed in India to one where Caste system brought on the rigidity that it did – was one where some Masters left their students without their work fully completed. Whether it was due to their lack of effort or due to unforeseen attacks from outside the country, or communal wars where the adversaries found nothing wrong in killing Spiritual Masters (like Guru Tegh Bahadur was by Aurangzeb). The origin of rigidity of the “Knowledge Class” was in all probability due to this missing link.
Caste system and its rigidity was a caricature of the Rigor required to pass the Spiritual knowledge in committed manner.