I am currently conducting a brain network analysis on my diffusion MRI data, using the connectomestats TFNBS algorithm. The final results that I get after running TFNBS are: beta0, beta1, enhanced, fwe_1mpvalue, null_contributions, null_dist, tvalue, uncorrected_pvalue, Zstat, abs_effect, cond, std_dev and std_effect.
Out of these results, I gathered that the relevant subnetwork could be identified using the fwe_1mpvalue file. Because having identical FWE-corrected p values demonstrate catagorisation by emergent effects in TFNBS, I was wondering if the edges represented by different, discrete p-values in the fwe-1mpvalue file can be separated out as independent subnetworks. For example, will all the edges with a p value equal to 0.0002 be a part of Subnetwork A, while all the edges with a p value equal to 0.0004 be a part of Subnetwork B?
Or, would it be more statistically accurate to look at the aggregate of all edges with a fwe-1m pvalue under a certain, higher threshold such as p<0.05? As an extension of the previous example, will all the edges with a p value equal to 0.0002 and 0.0004 be aggregated together into a single subnetwork, under a given p value threshold?
Furthermore, if we were to use TFNBS results in predicting the presence of a network correlating with a certain variable, would it be possible we use more statistically significant fwe-corrected p value thresholds such as p < 0.0005?
The TFNBS algorithm assigns a uniquep-value to each edge. Those p-values are influenced by both the test statistic quantified for that specific edge, and the test statistics of other edges in the network that provide support as per the TFNBS statistical enhancement algorithm. But those values are determined individually for each edge. You cannot make any inference about “connected sub-networks” based on equivalent p-values.
What may be somewhat misleading you is the fact that the p-values resulting from non-parametric permutation testing are discrete. If the TFNBS-enhanced test statistic of any individual edge exceeds that of the maxima observed in any random permutation, then the resulting p-value will be 1/5000 = 0.0002 for 5,000 permutations. So if you have multiple edges for which p=0.0002, all that tells you is that the enhanced test statistics in all of those edges is very large; it doesn’t actually tell you anything about whether or not those edges form a sub-network of any kind. Indeed if you performed a sufficiently large number of permutations, and used a sufficiently small value of dh for TFNBS, I would expect this false equality of p-values to break down, and you would start to see unique p-values in every edge.
The only appropriate interpretation of the data specifically following TFNBS (since the interpretation differs from that of NBS) is to compare the calculated FWE-corrected p-values to your pre-specified alpha value (typically 0.05), and for each edge individually, determine whether or not that edge reached statistical significance. Interpretation of any higher-level patterns in the data (e.g. multiple edges with common nodes that survive the p-value threshold) is subjective and not explicitly justified by the data (even though the connected-ness of those edges was exploited for the purpose of statistical enhancement).
While you have the ability to apply any p-value threshold you choose (and this is why we output the raw p-values and not a “significance mask”), from a scientific perspective you do not want to be experimenting with different p-value thresholds to see which result you “like most”. The alpha value corresponding to the statistical significance threshold should be specified before the experiment is performed.