Now I can't remember exactly where in the book this is noted (I will settle down on some cozy night with it and try to find the reference), but if I remember correctly it is one of the Master Face Dancers themselves that strongly imply that they have accumulated the knowledge of all the personas they have even become, and in this way achieved an understanding greater than the sum of the individuals they have impersonated.
This is a recurring theme - the BG is based entirely on Reverend Mothers being able to access the cumulative personalities and memories of their ancestors - the the last book the explanation given for Murbella's understanding of the BG Way is that she could witness the horror of the past through the Voices Inside, which is the common understanding that binds all BG and allows them to trust one another. Both Paul and Leto II had access to this ability, in addition to prescience; hence their devestating effect on the Universe. The final books also note that the Tleilaxu achieved their own immortality and ability to gather memories of many lives by using their own cells to recreate themselves in their Tanks.
Duncan's abilities go one step further - and he muses on this point more than once. He is aware that cells could not have been retrieved from all of his lives, yet he has unbroken memory from his original birth onwards. It seems that he has gained abilities not only from the knowledge accumulated in many lives, but also has some ability to "fill in the gaps". Whether this ability is prescience or not is a good question: it would allow him to "fill in the gaps" and also explain his foresight that Marty would be waiting to ensnare them on their escape from Chapterhouse (although this could have been foreseen without prescience). On the other hand, Duncan is not of the Atreides bloodline, the only line known by the BG to display prescient ability. What we do know is that with each incarnation the Tleilaxu subtly altered Duncan's genetics ... it is possible that they introduced the Atreides prescience without the BG noticing.
I would disagree that the Master Face Dancers were created by Tleilaxu in the Scattering, as the Tleilaxu are consistently shown to be masters of control, and paranoid about control. They are also extremely loyal to other Tleilaxu (shown by the number of secrets they hid as an entire race for millennia). The Master Face Dancers seem to have overcome all conditioned controls, even the whistling language. This would mean that they are either free of the Tleilaxu, or there are Tleilaxu in the Scattering intending to destroy the Tleilaxu in the Old Empire. Given the loyalty of the Tleilazi and the disdain the Master Face Dancers apparently show for them, I would prefer the former option.
I believe the book suggests that by the accumulation of far more knowledge than a single person should have, the Master Face Dancers were able to overcome the restrictions placed upon them by the Tleilaxu, and take technology bounds into the future. Remember that while all other races/groups at the time have only historical knowledge, the Face Dancers could accumulate current knowledge from a vast number of people - arguably a powerful base from which to advance technologically. As an alternative, Face Dancers in the Scattering may themselves have created the Master Face Dancers, using their accumulated knowledge, in an attempt to free their "kind".
"Paired opposites define your longings"
This world is based on relativity, things exist only in relation to each
other. So do you.
There is you, and there is not you.
In this case I would define longing as non-physical need. To long is to try
to get and own something that is not you.
"Those longings imprison you"
Owning other things is an illusion. You cannot have what you long for. While
there is need, the need cannot be fulfilled. You don't get things by wanting
and needing, because the very act of longing for them places them beyond
your reach (you let the universe outgrow you!).
If you say you want this, it means that you don't have it, and that's the
reason for wanting it. The more you want it, the more you don't have it.
In other words:
Needs imprison you. They say: "You must have this, you must do this. You
can't have this and you can't do that". It limits you choices, it limits
your freedom, it makes you "less" than you are.
It seems like a paradox, but it is not. The answer is:
"Stop trying to hit me and hit me!" - Morpheus
"Just do it!" - ..
Needs are fears of things that may happen if we don't fulfill our needs. Release your fears. Be afraid of nothing. Don't accept what I said earlier about "There is you and there is not you". It limits you, there is no such thing as not you! That's why it is a whip. It whips you out of this universe into a magic one where anything can happen. It goes beyond words, beyond normal thinking, beyond what you define as yourself. Do not act because of need, but because you choose so!
My personal favorite is the Amtal Rule: "To know a thing well, know it's limits. Only when pushed beyond it's tolerances will true nature be seen."
And the next step is? ...
Some things I feel are missing from your discussion.
Which brings me nicely onto point 2.
Or Using KH powers as Leto II had, they had predicted that this would be the no-ship that Duncan would escape from Gammu in & had fitted the 'net' in it and adjusted it to deliver Duncan to them.
(Remember that LetoII wanted Duncan to mate with Siona, IMHO, the sandworm not eating Sheeana was his way of recognising the second chance & bringing to the BG's attention, who rightly tried to mate Duncan with her 'again')
Duncan escaping with Sheeana/Siona means Humanity has a chance.
[I have ignored the Master, Jews, Sandworm (NB Sandworm, not Trout-> no second metamorphesis &? spice withdrawal for Sheeana) & Futars because I agree with your summation that these are side-plots, P.S. I never understood the need to feint the attack at Gammu, except to rescue people who they didn't know were there!]
I think that the last three books are very much like the first three books - they are almost the same story, told over again with new names and details. The first three are about the Atreides' attempt to (in Leto II's words) "wrest an Empire from a ruling monarch", and then to defend that Empire against other factions who would fight for ascendancy. The last three books are about the same thing - a fight for ascendancy, featuring the modern Atreides, the Tleilaxu, Ix, Guild, etc. All the old players + their evolved offshoots from the Scattering.
What is familiar is that all of the old guard have once again failed to recognize the evolution of society as a factor in their designs - like the Sisterhood did when they produced Paul a generation too early and lost control of him. Modern BG, BT, Guild, and Ix has been grappling for control of the old Empire, while the people of the Scattering have evolved their own ascendancy plans and are fleeing a new threat (Wouter's 'Master Face Dancers' and their handlers, futars, and disease vectors). The people of the Scattering play the Jessica role - the unexpected thing.
Given this set - up, I think that Frank had two general choices with book 7.
One, to create a new Kwisatz Haderach, most likely the child of Murbella and Duncan, or Sheena and Duncan, but also possibly young Miles and someone new. The only thing I'm sure that Frank wouldn't do with the new KH is allow him or her to be the product of any faction's design - the essential theme of all the books is that the evolution of society invariably toys with any human pretensions to long-term planning (remember the Tyrant called himself the only truly long range human planner - and he was talking about more than just his 3500 year life-span).
Two, uphold Leto II's prescience and planning by denying ANY group ascendancy. Remember, It's not just the Master Face Dancers who have an unexpected ability, Duncan can also see them and their net and avoid it - their abilities are self cancelling. A book along these lines would be necessarily difficult to resolve, as there could be no really final resolution. While I'm sure that Frank had the ability to write an intelligent and engaging story along these lines, giving us some real surprises as new plans are cancelled by unexpected talents and dependencies, I don't think that Brian can pull it off (although I love the Dune universe so much that I'd read his attempt).
If Frank wanted another Kwisatz Haderach, maybe there should be a seventh book. If, however, he wanted to continue the Golden Path, I think the sixth book is a fine way to end it. It ends with a beginning, just as the Tyrant wanted, and the future will be just like the past, only different.
I agree entirely that the 7th book would have been focused on Duncan and Sheeana. I would like to point out that the bitter medicine of Sheeana that Murbella predicts was actually described in Chapterhouse and in Heretics. The Bene Gesserit planned to use Sheeana to regain religious control of much of the known universe. Sheeana's escape means that, as a fully qualified Reverend Mother, she will be able to take control of those religious forces on her own if she wishes to do so.
I find that a 7th book would have to be based on Duncan, Sheeana, and Teg, because they are the three humans with the most potential and power. While the survival of the Golden Path would mean that the three could not take over the universe, along with Scytale and his nullentrophy capsule, they represent a large chunk of the power in the known universe. I'm led to believe that Dune 7 might actually be an example of a reoccurance of the Dune history: Duncan and Sheeana starting a new society in a far off galaxy untouched by the Scattering, and starting a new Bene Gesserit, Sctyale remaking the Bene Tleilax, Duncan founding a new order of Mentats, etc. Eventually, a new Kwisatz Haderach would be produced, and take control of this galaxy much like Paul Atreides did. It just seems a fitting end to the Dune series, doesn't it?
The end of Chapterhouse seems somewhat like the end of God Emperor of Dune, though. Sheeana and Duncan Idaho escape the powers that be, with a very likely future pairing that will result in a new breed of humanity, just like the Siona/Duncan pairing resulted in humanity's partial immunity to prescience. It feels much the same, at least to me.
I think that Duncan Idaho was the Tleilaxu's response to the Master Face Dancers. After all, he was able to understand the net and destroy it. Duncan is, to me, the most interesting character in the series. He's in every book, with varying abilities, and he seems to be an important person to the universe. Leto II seemed to have believed so, at least.
One more thing: I don't think the Golden Path is going to be ended in the 7th book. Herbert's vision is definitely one of the potential of humanity as opposed to its downfall, so that just seems incredibly unlikely.
Well, I thought I'd offer my thoughts. Here's to Brian Herbert and Kevin Anderson not ruining yet another book for all of us Dune fans out there.