Analyzing tskgen/SIFT goodness of fit with SIFT2

1

Dear All,

The SIFT2 paper proposes that one may use SIFT2 on the output of tskgen/SIFT to improve recons, to be tested in the future…
I wanted to see if there is an easy way to use SIFT2 outputs when SIFT2 is applied to the tskgen /SIFT output (tsk file) to test the quality of the latter. In the paper, for a given regularization parameter (lambda, say reg=0.1) value, this would mean the data residuals x (eq 11). A lower x for a particular recon would mean better recon (leaving out the issue of one measure capturing the whole phenomenon etc).
As implemented currently, PM, FD in the formula etc are not available as outputs. Is there a way to do this just by using mu and the streamline_coef csv file. Intuitively, a ‘perfect’ tckgen/SIFT recon would generate a streamline_coef csv file with all coeffs=0. One may think of generating mean and var (or interquantile range etc) on the product mu* streamline_coefs file to compare tskgen/SIFT recons, say when testing within the same subject (native vs upsampled fod file, different seeding options etc), with the idea that lower mean or dispersion values would reflect better recons, please advise.

Octavian

2

Any thoughts on this? Again, it follows that if tckgen/SIFT does a good job, the SIFT2 (applied to tckgen/SIFT output) should generate streamline coefficients tending to 0, and this could be a criterion to judge tckgen/SIFT, is that right? Thank you,
Octavian

3

I suppose that should be “tending to 1”. Or more generally probably “tending to the same number among them” (but I’m not sure if SIFT2 does some kind of normalisation, or allows the coefficients as a whole to “drift”).

You do have to be careful when saying…

…up to a certain extent. To add some nuance: it judges the performance of SIFT, but also “how you set it up”. A part of the latter would for instance be how many streamlines were present in the original tractogram, because that may already limit the discrete space that SIFT can play with (if not enough streamlines, or a low number in general). Even if SIFT does a perfect job, it’s still only giving the best possible solution in the discrete space you’re allowing it to. SIFT2 in a way goes beyond the discretisation limit by allowing continuous weights.

But given enough streamlines (so some kind of massive number, that is), it would indeed be interesting to see what SIFT2 “does” with SIFT output. But when I say “massive” number, make it extra-massive, so even the SIFT output still has a “massive” number for SIFT2 to play with.

4

The SIFT2 paper proposes that one may use SIFT2 on the output of tckgen/SIFT to improve recons, to be tested in the future…

I’m not sure I would be so bold as to say that it could “improve recons”; it’s simply a way in which these commands could plausibly be used. I had little doubt that this sort of combined approach would seem attractive to many people, indeed many actually criticise SIFT2 for retaining all streamlines since it “no longer has the capacity remove false positives”. The reality is that this is a misinterpretation that I’m seeing more and more often: Even though SIFT can remove streamlines, that doesn’t mean it’s actually specifically removing false positives, it’s just reducing the streamlines density in pathways that are over-reconstructed. So whether or not there’s a benefit to throwing out streamlines as opposed to just letting them obtain very small weights: Maybe, since it’ll let you bump up the regularisation coefficient without having those “unwanted” streamlines contribute more than you want, but there are a lot of complex interactions at play.

Not sure whether quantitatively assessing the SIFT / SIFT2 combination in this fashion is going to work. You’re probably looking at Eq. 5 rather than Eq. 11, the latter is specifically for generating L-curves and doesn’t properly scale the regularisation term to match the data term (Eq. 6).

As implemented currently, PM, FD in the formula etc are not available as outputs.

Try the -output_debug option :stuck_out_tongue: The -csv option may be handy to you as well.

One may think of generating mean and var (or interquartile range etc) on the product mu* streamline_coefs file to compare tskgen/SIFT recons, say when testing within the same subject (native vs upsampled fod file, different seeding options etc), with the idea that lower mean or dispersion values would reflect better recons, please advise.

As you say, ideally you’d have all coefficients = 0; but you’d also have the data error term being zero. Ultimately neither of those is going to be the case, and hence you need some mechanism by which to “combine” these two sources of “error” into a single metric. Otherwise, if one is bigger than your reference and one is smaller than your reference, there’s no way to say whether or not it’s “better” or just different. This is precisely what the cost function in Eq. 5 does.

It’s possible that some other metric on the distribution of streamline coefficients may be more useful in terms of subsequently “assessing performance” of the algorithm, but I never found the metric I was looking for in that respect.

streamline coefficients tending to 0

I suppose that should be “tending to 1”.

The streamline weighting coefficients will tend to zero, which corresponds to a streamline weighting factor of 1. It’s the latter that gets output from tcksift2, but all of the optimisation is done with the former.