Eeek… that’s an awkward one. Let’s see if my interpretation here is correct:
Streamlines can propagate and terminate entirely outside the termination mask.
Streamlines can propagate and terminate entirely within the termination mask.
Streamlines can propagate from outside the termination mask into the termination mask.
Upon entering the termination mask, streamlines are still free to terminate at any point within that mask due to other streamlines tractography criteria.
If a streamline attempts to propagate from inside the termination mask to outside the termination mask, it is terminated due to the constraint imposed by having provided that termination mask.
If that’s the case, the closest thing I can suggest would be to define a 5TT image where what you define as a “termination mask” is labelled as sub-cortical grey matter. Streamlines will be forbidden from entering and then exiting that region by ACT prior #6. (They will however be truncated and so the termination may no longer be exactly at the point where the streamline attempted to exit the mask, but this is something that could be temporarily disabled short-term / controlled by a command-line option flag long-term.
Otherwise, it’s a slightly clumsy functionality to even define appropriately; so I’d be curious to know whether this is simply a feature that can be used to serve your purpose that you’re looking to duplicate in MRtrix3, as opposed to having a purpose that needs serving but could be achieved in some other manner using either existing or more appropriate methods.