FBA design / contrast matrices for three groups

Olga · July 26, 2018, 6:13pm

Hello experts!

I’m trying to compute FBA, and would appreciate your advice on statistical matrices construction. We have three groups: typical population (T), disease population (D), and disease-related (Dr) population. My first version of design matrix is this:

1 1 0 0
1 0 1 0
1 0 0 1

Where the first column is intercept, the second is T group, the third is D group, and the fourth is Dr group. N lines for N subjects.
The contrast matrix for me to see the difference between groups:

(1) D > T 0 -1 1 0 (2) D < T 0 1 -1 0
(3) Dr > T 0 -1 0 1 (4) Dr < T 0 1 0 -1

Now comes the puzzle. Since fixelcfestats only accepts a single contrast, will I need to run the analysis twice? First, I will run (1) and (2) (supplying (1) and requesting -negative output as well), and then run (3) and (4) in the same manner? My question is whether this would be a correct way of running this type of analysis.

Thank you!

Olga

rsmith · August 15, 2018, 4:53am

Hi Olga,

This design matrix may misbehave as it is rank-deficient: the sum of columns 2-4 is equivalent to column 1, and therefore there are an infinite number of solutions for the beta values. While the GLM code is designed to still work in such circumstances, and due to the nature of your intended contrasts it shouldn’t actually have an effect on your t-values, it’s nevertheless avoided as best practise.

You could simply omit the first column entirely:

(for two subjects in each group)

Moreover, while you have three groups in your experiment, none of your hypotheses actually require all three groups to be present in the model at the same time; and fixelcfestats can’t (currently) test more than one hypothesis at a time. So you could simply construct two separate design matrices: One for comparing T and D, one for comparing T and Dr; then run fixelcfestats separately for each.

Rob

Olga · August 16, 2018, 4:51pm

Hi Rob,

Thank you! That’s what I ended up doing. Watched a bunch of Jeanette Mumford’s brain stats videos, and read a lot of GLM theory…

If you don’t mind, can I piggyback on here with a couple of questions? First, I’m a little pained by the FWE correction. Is there an option to use FDR instead?

Second, may be a little silly question, but being new to these processes I’m trying to gain a thorough understanding of it all. I ran a simple correlation using a single group to see if some behavioral measures associate with WM “integrity” (for lack of a better word…). The results look interesting, but I could not wrap my head around what the absolute effect output would mean in this scenario. I “discovered” it’s identical to my beta of interest. Am I understanding it right that it just is designed to create this output?

Thanks!

rsmith · August 17, 2018, 1:19am

Watched a bunch of Jeanette Mumford’s brain stats videos, and read a lot of GLM theory

I’m sure there’s plenty of people on here interested in better understanding the GLM - and I don’t really want to spend an inordinate amount of time writing documentation on it - so if you can provide links to what you found useful I’m sure I’m not the only one who would be appreciative

I’m a little pained by the FWE correction. Is there an option to use FDR instead?

Currently no. I suppose it would technically be possible, but would require someone motivated enough to do it.

… I could not wrap my head around what the absolute effect output would mean in this scenario. I “discovered” it’s identical to my beta of interest. Am I understanding it right that it just is designed to create this output?

The “absolute effect” is the inner product of the beta values with the contrast vector. So if your contrast is a single “1” and zeroes in all other columns, then yes it is identical to the beta value of that column. In certain circumstances it can be used in certain calculations and/or have a certain interpretation, but it’s best understood from the underlying fundamental mathematics.

Olga · September 6, 2018, 1:40am

Rob,

Thanks for detailed response!

Here is the youtube channel which is very helpful in my statistical inquiries. https://www.youtube.com/channel/UCZ7gF0zm35FwrFpDND6DWeA

I hope people find it useful.

Thanks,
Olga

silver_mark · September 20, 2020, 7:35am

Hi,Olga
I’m glad to see your post when I’m wondering how to set design_matrix and Contrast_matrix.
So,in the end,because of my poor comprehension ability, I didn’t know how to set design_matrix and Contrast – matrix.Can you give me some advices?
Or,a concrete example is given(Group A:001_control,002_control,003_control;Group B:001_patient,002_patient,003_patient),how to set this example’s two matrix?
Last question, How to implement FWE correction?

Thanks!
Silver

rsmith · October 1, 2020, 1:10am

Hi Silver,

From your brief description it looks like you’re doing just a basic two-group difference?

Last question, How to implement FWE correction?

This is not something that you yourself need to “implement”; the fixelcfestats command performs FWE correction and provides you with FWE-corrected p-values.

Rob

silver_mark · October 6, 2020, 8:06am

Hi Rob,
I’m glad you can answer my question.
Yes,I’m looking at FIXEL changes in the same group of patients at different points in time,so I use paired T-test.At the samt time,I am also examining FIXEL differences between this patient group and the healthy group (inconsistencies in number),so I use unpaired T-test.

NOW,I have a question:
Shouldn’t the FWE corrected P value appear after the uncorrected P value?But why is it now that the corrected P value appears first?

Silver

rsmith · October 8, 2020, 5:28am

I’m looking at FIXEL changes in the same group of patients at different points in time,so I use paired T-test.At the same time,I am also examining FIXEL differences between this patient group and the healthy group (inconsistencies in number),so I use unpaired T-test.

These are two very different experiments; and it’s not just the design matrix that differs between the two, but also the nature of the model errors that determine how shuffling should be done. So if you’re still unsure how your GLM should be set up across multiple experiments, make sure that you describe all such experiments.

Shouldn’t the FWE corrected P value appear after the uncorrected P value?But why is it now that the corrected P value appears first?

What you’re looking at there is purely the time at which the file is first created on the filesystem, which is not guaranteed to be in the same order as that in which they are computed, depending on exactly how the code is written. In this case, you can indeed see in the code that uncorrected_pvalue.mif is written after fwe_1mpvalue.mif.

However I’m going to infer from your question that you were in fact expecting the former to be written a long time before the former, since it “doesn’t require FWE correction” and therefore should have been calculated and written a long time before the FWE-corrected p-values. However the uncorrected p-values still require computing all shuffles of the data. The uncorrected p-value is the fraction of shuffles for which the enhanced test statistic in that fixel was greater than that in the unshuffled data; whilst this may be independent of all other fixels (and hence “uncorrected”), it still requires performing all shuffles. Hence why they’re written to the filesystem at approximately the same time.

Rob

silver_mark · October 12, 2020, 2:32pm

Hi,Rob

I’m sorry to bother you with these simple questions.Thank you very much for your kindness to answer my questions.
OK,I will describe my experimental setup in detail.
I have data on 37 patients at four different points in time.Like:

and I have 15 healthy controls data at basedline.
I want to compare 37 patients pre-therapy data and 15 controls data(I use unpaired two-sample T-test).What’s more,I want to compare the 37 patients at each point in time, and I want to do a pairwise comparison(like,pre-therapy vs post-treatment,post-treatment vs 3 months later…6 conditions in total) so I am going to do the paired two-sample T-test 6 times.
Take an example of a situation:(pre-therapy vs post-treatment)(I just take two patients’ data at two time as example,like:patient1:data1.1 data1.2 ; patient2:data2.1 data2.2 )(FD comparision)

design_matrix:
1 0
1 0
0 1
0 1

contrast_matrix:
1 -1

I know that with so little data, the results are not statistically significant.I just want to familiarize myself with the steps of the analysis process and the setup of the design and contrast matrix.

So,I get these results:

I know what the uncorrected_pval.mif means and the fwe_1mpvalue.mif represents the FWE corrected P value.Actually,These two P values are (1-P).

And I mrview wmfod_template.mif and use fixel plot to overlay fwe_1mpvalue.mif.
I have some questions about this:
1.The image 1 shows the results I got using the MRView command directly.
2.Do files with “Colour by”, “Thersholds” and “Scale by” boxes have to be the same?(Like image 2)
Should the minimum threshold of 0.95 be set this way?(Like image 2)

Lastly,I would like to show significant differences in streamlines according to FIXEL with significant differences, is there any relevant tutorial?(I have performed whole-brain fibre tractography on the FOD template and performed SIFT to reduce tractography biases in the whole-brain tractogram)

Silver

rsmith · October 14, 2020, 5:39am

I want to compare 37 patients pre-therapy data and 15 controls data(I use unpaired two-sample T-test).

Yep, that’s fine; commonplace two-sample unpaired t-test.

What’s more,I want to compare the 37 patients at each point in time, and I want to do a pairwise comparison (like,pre-therapy vs post-treatment,post-treatment vs 3 months later…6 conditions in total) so I am going to do the paired two-sample T-test 6 times.

So there’s a couple of points to raise from here. Neither of which are actually specific to your experiment, but given you’ve shown your setup I think it’s a good exemplar upon which to explain to a wider audience.

While you can do as you’ve described, it’s worth interrogating whether or not it is in fact what you want to do (if fully informed of the scope of possibilities). What I’m going to be poking and prodding at here is whether or not there has appeared a divergence between the processing that you intend to apply to your data, and the fundamental hypotheses you want to test that necessitated the collection of those data.

For instance, the fact that you have data immediately before and after “treatment” suggests that you believe that there may be some immediate change in the image data. If so, then a one-sample t-test on the difference between pre- and post-treatment makes sense.

For the follow-up scans, it’s not quite so obvious. Based on your current experimental proposal, particularly since you stated that you were going to test 6 conditions in total, the hypotheses you would be testing are:
1. There is an observable difference between pre-treatment and post-treatment.
2. There is an observable difference between pre-treatment and 3 months.
3. There is an observable difference between pre-treatment and 6 months.
4. There is an observable difference between post-treatment and 3 months.
5. There is an observable difference between post-treatment and 6 months.
6. There is an observable difference between 3 months and 6 months.
Now these are entirely valid hypotheses to test, but I suspect that when you first devised the experiment, this is probably not what you described in the project description document.

Your original hypothesis might have been more like “there is an observable difference over time post-treatment”. That’s a more intuitive hypothesis conceptually, harder to implement practically, but it can in fact be done with sufficient GLM competency.

You could even generalise further and say that your hypothesis is more like “There is an observable difference from the treatment”, without specificity of when that difference manifests for the purpose of statistical inference (one can always interrogate the data post hoc to see where the effect arose). This is also achievable with GLM trickery.
Something that’s been bugging me for a long time is the issue of false positive control in FBA, and it stands out particularly in your proposal.

Many will know that performing statistical inference using Connectivity-based Fixel Enhancement (CFE) provides Family-Wise Error (FWE) control, which is more stringent than FDR and more powerful than Bonferroni. However the story is actually a little more complex. What CFE provides (by default) is what’s known as weak FWE control. This is where, for each hypothesis test individually, you have a e.g. 5% likelihood of reporting a false positive anywhere; but each hypothesis is treated entirely independently.

So what’s the problem? Well, let’s take your example, and let’s say hypothetically that you also want to test the three fixel-wise metrics in the default FBA pipeline (FD, FC, FDC). First, you do a (pre-treatment) patient-control comparison; that’s three independent hypotheses. Now you do the six tests enumerated above, for each of the three fixel metrics; that’s another 18 tests for a total of 21 hypothesis tests. Now, while each of those tests individually has a e.g. 5% chance of reporting a false positive, across the gamut of your entire experiment, your chance of reporting at least one false positive is more like:

1.0 - (0.95)²¹ = 66%

So one of the benefits of the alternative hypothesis construction I described above is that it reduces your total number of hypothesis tests, which reduces the magnitude of this issue.

There are also other ways in which you can modify an FBA experiment in order to alleviate this issue. One I’ll mention here is the prospect of utilising strong FWE control, where one performs a suite of hypothesis tests with only a e.g. 5% chance of reporting a false positive not only in any location in the brain, but also in any of a number of hypothesis tests. This can be done using parametric testing using the -strong option in MRtrix3 commands; this manuscript also provides justification for doing such in this way. Another proposed trick that is specific to FBA has been used in a couple of manuscripts that are currently under review, I can hopefully show that one in the near future. I have in mind multiple potential solutions to this issue, unfortunately I’ve not been able to perform the barrage of tests across multiple experimental datasets that would be necessary in order to make a concrete recommendation and change the documentation default pipeline…

2.Do files with “Colour by”, “Thersholds” and “Scale by” boxes have to be the same?(Like image 2)

Definitely no. It’s very intentionally designed in this way. This is a strong benefit of the fixel directory format, as mrview immediately provides the drop-down list of files available for modulating the visual properties of each fixel.

Should the minimum threshold of 0.95 be set this way?(Like image 2)

Yes, that’s correct; though colouring fixels by p-value is less common, and modulating fixel length based on p-value is less common again.

Lastly,I would like to show significant differences in streamlines according to FIXEL with significant differences, is there any relevant tutorial?

https://mrtrix.readthedocs.io/en/3.0.2/fixel_based_analysis/displaying_results_with_streamlines.html

Cheers
Rob

silver_mark · October 15, 2020, 3:51am

Yes.Thank you for your patient and detailed answer.
Undoubtedly,I know how to set design matrix and contrast matrix when I want to achieve two-sample unpaired T-test.

Yes,what I want to do is:
1、Compared patients(pre-treatment vs controls) using two-sample unpaired T-test in fixel.Like this image(Page 7 in this article)

2、

The six situations you listed are exactly what I want.At first,in order to check whether there is a significant difference between the pairings of the four time points,I want to do do a two-sample paired T-test for each of these six cases.
But When I see “Tripled” T-test,I wonder if there is “quadruple” T-test(Because I have data at four different times).If it does exit,What will its design_matrix and Contrast_matrix look like?(Take “5 subjects,each scanned under 4 conditions” as example)

“5 subjects,each scanned under 3 conditions” (design_matrix and Three Contrast matrix items selected for specific situations)

3、I have a problem(Same kind of problem) showing the FWE corrected P value and P value.When I show fwe_1mpvalue.mif on wmfod_template.mif,I know that these two are in the same range with the red coils as is shown in the first diagram below(I am not setting Threshold).When I set the minimum Threshold as 0.3(I know it should be 0.95 if I want see P value is less than 0.05),The range of colourbar displayed values has changed to a minimum of 0.66666687(as is shown in the second diagram below).I don’t know why this happened.AND,What range of values does the white coil represent?

4、I use two subjects and their data on two different time to "found significant differences in streamlines according to FIXEL with significant differences ".But no streamlines with any significant changes are shown.Maybe it’s because the sample size is too small?

I encountered too many problems along the way .Sorry for bothering you so many times.

Silver

rsmith · October 21, 2020, 2:44am

But When I see “Tripled” T-test,I wonder if there is “quadruple” T-test(Because I have data at four different times).

Those instructions are specifically for FSL feat; for FSL randomize, to which fixelcfestats is more closely related, the page states that such a test is not possible in an integrated fashion. Even if it were, I think that compound symmetry would be violated in your case, since the variance between pre- & post-treatment is likely different to the variance between post-treatment and 3 month follow-up.

When I set the minimum Threshold as 0.4(I know it should be 0.95 if I want see P value is less than 0.05),The range of colourbar displayed values has changed to a minimum of 0.66666687(as is shown in the second diagram below).

I’d need to search through the code to confirm, but I believe what’s happening here is that the manual lower colour value (0.4) is mapped to the start of the colour map (black in the case of the hot colourmap); but when the colour map overlay is drawn, it only shows the visible portion of the mapping. So if, in your p-value image, there are no fixels that possess a value in between 0.4 and 0.667, then once you engage the lower threshold of 0.4, mrview determines that there is no purpose in drawing the portion of the colour map in between 0.4 and 0.667 because there are no fixels present with those values, and so “zooms in” on a subset of the colour map for the purpose of the overlay.

What range of values does the white coil represent?

This is the colour map of the main mrview image, i.e. the WM density template image in this case. So -0.006 maps to black, 0.0 maps to a very dark grey, 0.396 (and greater) map to white. A value of ~ 0.282 (1/sqrt(4pi)) corresponds to a fibre density equivalent to that of the response function.

I use two subjects and their data on two different time to "found significant differences in streamlines according to FIXEL with significant differences ".But no streamlines with any significant changes are shown.Maybe it’s because the sample size is too small?

I’m hoping that in doing so, the command will have issued a warning that based on the sample size it was unable to achieve the default number of 5000 permutations, and would instead utilise all possible permutations. In your case, the maximum number of permutations is 2. Therefore the smallest p-value that can be mathematically obtained is 0.50.

Cheers
Rob

silver_mark · November 4, 2020, 1:41am

Hi,Rob
Thank you very much for your patience.My code has run through, but the end result is not particularly good.Significant differences in streamlines were rare.So I wrote another post asking a few questions.I hope you can give professional advice when you read this post.Finally, Thank you very much for your help.

Cheers
Silver

OOjinaga · October 19, 2021, 2:34pm

Hi @rsmith,

This discussion thread has been extremely helpful to create my design and contrast matrices for a FBA.
I just wanted to check if they are correct.

I am doing a between group comparison (controls (n=39) vs. patient (n=10)), controlling for age and sex. This is part of my my design matrix:

1 -0.01 0 1
1 0.99 1 1
1 -3.7 1 1
-1 -5.44 0 1
-1 0.99 0 1
-1 -3.73 1 1

Where the first column is group (1 controls, -1 patients), then age in weeks (demeaned), then sex and lastly the intercept.

I would like to test the two hypotheses with the following contrast:

controls>patients 1 0 0 0
controls<patients -1 0 0 0

I am not sure about the contrasts. Also, does fixelcfestats still not test more than one hypothesis at a time, or can I have both contrasts in the same contrast.txt file?

Many thanks,
Olatz

rsmith · October 26, 2021, 12:06pm

Welcome Olatz!

I just wanted to check if they are correct.

All seems fine as far as hypothesis testing is concerned. There are however consequences to be aware of if doing anything that involves interpretation of the raw GLM beta coefficients.

Also, does fixelcfestats still not test more than one hypothesis at a time, or can I have both contrasts in the same contrast.txt file?

As of version 3.0.0 (relevant post in changelog), the contrast matrix can have multiple rows, and each is used to define a stand-alone hypothesis.

Cheers
Rob

OOjinaga · October 26, 2021, 3:41pm

Hi Rob,

Thank you for the support.

What would be the consequences to be aware of if I were to do anything involves interpretation of the raw GLM beta coefficients?

Best,
Olatz

yichao · December 30, 2021, 7:10am

Hi,epxerts,
I am a green hand. I am trying to the relationship FD,FC, FDC as the age changing. for example. I have just one group: the control01,age:38, control02,age:40,control03,age:45, control04,age:50,…
I do not know how to set design_matrix and Contrast_matrix.
Can you give me some advices? @rsmith @silver_mark

rsmith · January 7, 2022, 6:50am

Hi @yichao,

In the most basic scenario, where you want to test the hypothesis that the rate of change of e.g. FDC with respect to age is greater than zero, and don’t want to bother with parameter normalisation, you would do:

Name      | CI | Age
----------+----+----
Control01 |  1 | 38
Control02 |  1 | 40
Control03 |  1 | 45
Control04 |  1 | 50


t-test:      0 |  1

If there is no relationship between age and FDC, i.e. the null hypothesis is true, then the second GLM beta coefficient will be zero. If you want to test the hypothesis that FDC decreases as a function of age, i.e. the second beta coefficient is negative, then the contrast vector [ 0 -1] would produce positive t-statistics in that instance.

Cheers
Rob

Hinke_Halbertsma · February 9, 2023, 9:30am

Dear @rsmith,

I came across this thread because I want to do something similar, although it doesn’t exactly answer my question.

Some background: I’m working on an extensive data set with both patients (n=160) and controls (n=640). For each of these subjects, we’ve calculated Polygenetic Risk Scores (PRS) for two diseases (D1 and D2). Based on their PRS, we’ve grouped the subjects into four groups: D1high_D2high, D1high_D2low, D1low_D2high, and D1low_D2low. I hope this makes sense.

In each of these groups, we have 160 controls and 40 patients. So the first few lines of the design matrix should look like the one attached right? NB: for clarity, I’ve included a controls/patient header and omitted the covariates. In the design matrix, I’ve stuck to the 4:1 ratio of controls vs patients.

Now my question is how I need to define my contrasts. We have some hypotheses:

Controls > patients in general, so then the contrast should be: 0.25 0.25 0.25 0.25 -0.25 -0.25 -0.25 -0.25. Am I right?
There is a difference between the 4 groups, where we hypothesize that D1low_D2low > D1high_D2high, with D1low_D2high and D1high_D2low in between (but no hypothesis on the order). Should I run separate tests for this? Any suggestions on how to do that?
There might be an interaction effect, where the effect described above is bigger for the patients than for the controls. Any suggestions on how to do that?

Is its feasible at all to do this with fixelcfestats?

Any help would be much appreciated!