# Suggestion for tckedit **URL:** \ **Category:** Uncategorized\ **Created:** [February 25, 2017, 2:20pm UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747 "2017-02-25T14:20:20Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![wamigy](https://community.mrtrix.org/letter_avatar_proxy/v4/letter/w/b5a626/32.png) [@wamigy](https://community.mrtrix.org/u/wamigy)\ **Post date:** [February 25, 2017, 2:20pm UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/1 "2017-02-25T14:20:20Z")
Hello Mrtrix devs’, First of all thanks again for your wonderful work and your efforts for an efficient implementation of your tools. Everything works like a charm. However, if I may, I would have a suggestion to reduce processing time especially when dealing with very large sets of fibers. I generate big .tck files (\>100M tracs) and, of course, I have to split the file for further processing for RAM constraints. I’m using tckedit do to so but it takes a while to generate smaller packets. Let’s say I have 100M fibers and would like to generate 100 packets of 1M fibers. I have to call 100 times tckedit playing with the option -skip and therefore, it requires scanning repeatedly the big .tck file (in total 99M+98M+97M+96M+… = 4950M fibers scanned without writing). Is there a way to split the 100M tck file into 100 smaller 1M tck files without scanning numerous times the original file ? If not, do you think an option like “-divide N” could be integrated to tckedit to split a tck file into N smaller files ? Thank you again! Best regards, W. ---
**Author:** ![rsmith](https://community.mrtrix.org/user_avatar/community.mrtrix.org/rsmith/32/2672_2.png) [@rsmith](https://community.mrtrix.org/u/rsmith)\ **Post date:** [February 27, 2017, 6:15am UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/2 "2017-02-27T06:15:20Z")
Hi @wamigy, Yes, that’s an entirely reasonable suggestion. If you’d like, you can post a new issue on [GitHub](https://github.com/MRtrix3/mrtrix3/issues) (or I can add it myself if you don’t have an account). Indeed if anybody has ever wanted to try their hand at contributing to _MRtrix3_, this would be a nice easy entry point! 🙃 I’m not sure whether such a functionality would go into `tckedit`, since it would introduce further ambiguity in processing order: e.g. If you want to split into chunks of 1M, but you also specify a `-include` ROI, should the files be split for each 1M _input_ tracks, or for each 1M _output_ tracks? It might be more at home in `tckconvert`, or maybe even its own command `tcksplit`. Though I will question this bit first: > However, if I may, I would have a suggestion to reduce processing time especially when dealing with very large sets of fibers. I generate big .tck files (\>100M tracs) and, of course, I have to split the file for further processing for RAM constraints. Apart from certain commands (`fixelcfestats`, `tcksift`, `tcksift2`, any others?), the size of the input track file should not actually significantly influence the RAM usage of the command: Tracks are processed as a _stream_ of data, and don’t need to be stored in memory in their entirety. Moreover, many commands will use multi-threading, so for a single system splitting the tracks across multiple files and running the same command multiple times won’t actually be any faster than running one instance with multi-threading. So splitting / processing / merging track files may even be slower… Is there a specific use case that you’re having performance issues with? Cheers Rob ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [February 27, 2017, 9:05am UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/3 "2017-02-27T09:05:55Z")
> [@rsmith](#): > > Apart from certain commands (fixelcfestats, tcksift, tcksift2, any others?), the size of the input track file should not actually significantly influence the RAM usage of the command: Tracks are processed as a stream of data, and don’t need to be stored in memory in their entirety. Moreover, many commands will use multi-threading, so for a single system splitting the tracks across multiple files and running the same command multiple times won’t actually be any faster than running one instance with multi-threading. So splitting / processing / merging track files may even be slower… Is there a specific use case that you’re having performance issues with? I fully agree with @rsmith here, and I’d add that the commands referred to as needing to load the whole tracks file (e.g. `fixelcfestats`, `tcksift`, `tcksift2`) do so out of necessity - they need access to all the streamlines (or data derived from them) during processing for correct execution. If the work could be parallelised or split in some way to reduce RAM usage, I think we would already be doing it (unless it _really_ massively deteriorated performance). So if you’re thinking of splitting the tracks files for use with any of these commands, you will get suboptimal results from merging the results, compared to processing the full tracks file as one (with all the RAM requirements that entails). Cheers, Donald. ---
**Author:** ![wamigy](https://community.mrtrix.org/letter_avatar_proxy/v4/letter/w/b5a626/32.png) [@wamigy](https://community.mrtrix.org/u/wamigy)\ **Post date:** [February 27, 2017, 9:54am UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/4 "2017-02-27T09:54:36Z")
Thanks for your answers. To be more specific, small packets are easier to handle with matlab for further processing. In my pipeline, this step would be the very last of all mrtrix commands (and yes I totally admit I might be the only one interested in such a feature 😉). I also understand introducing a function called `tcksplit` may be misleading for users. ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [February 27, 2017, 10:38am UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/5 "2017-02-27T10:38:02Z")
> [@wamigy](#): > > To be more specific, small packets are easier to handle with matlab for further processing. Maybe it would be better to modify or implement a different Matlab function to allow loading streamlines one at a time…? That should be relatively simple, we’d just need to split the current `read_mrtrix_tracks.m` function into 3 functions: open() / read\_next() / close() - maybe a candidate for [Matlab’s OOP functionality](https://uk.mathworks.com/help/matlab/matlab_oop/create-a-simple-class.html)? ---
**Author:** ![wamigy](https://community.mrtrix.org/letter_avatar_proxy/v4/letter/w/b5a626/32.png) [@wamigy](https://community.mrtrix.org/u/wamigy)\ **Post date:** [February 27, 2017, 1:24pm UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/6 "2017-02-27T13:24:57Z")
Ok yes you’re right, actually I’ve already written a matlab function for this task, this suggestion was “in case” you could see an interest or someone else could need it. Thank you again! ---
**Author:** ![jdtournier](https://community.mrtrix.org/user_avatar/community.mrtrix.org/jdtournier/32/2594_2.png) [@jdtournier](https://community.mrtrix.org/u/jdtournier)\ **Post date:** [February 28, 2017, 8:19am UTC](https://community.mrtrix.org/t/suggestion-for-tckedit/747/7 "2017-02-28T08:19:59Z")
Understood. Great to have suggestions like this, please keep them coming. In this case however, I think your own needs would be better addressed by modifying the matlab functions to allow the data to be streamed (much like most _MRtrix3_ commands already operate). Otherwise, there may be other legitimate use cases where splitting a large tractogram as you suggest might come in handy - if anyone has such a need, please make yourselves heard. It’s not difficult to implement, we’d be happy to if we felt there was sufficient demand.