For input files that fit in memory, the best way to get good performance is making sure that the tool can allocate enough memory to contain the file, and optionally increasing the number of threads.
For input files that do not fit in memory, the best way to get good performance is minimizing the number of required merge passes. This can be done by increasing the _branch factor_, and to a lesser degree by increasing the amount of memory.
## Status of this software
SortBin is just a hobby project, not production-quality software.
My limited testing suggests that the tool works correctly and is reasonably efficient.
Testing was done on Linux, but the tool should probably also work on other POSIX systems.
## How it works
If the input file fits in memory, SortBin simply runs an in-place quicksort on the whole file.
Optionally, multiple threads may be used to run the quicksort algorithm on parallel CPU cores.
If the file does not fit in memory, SortBin uses a sort-then-merge strategy:
- Create a temporary file with the same size as the input.
- Chop the input into blocks that fit in memory. Read each block, sort it with quicksort, then write to the temporary file.
- Read multiple sorted blocks from the temporary file and merge them into a bigger sorted block. By default, SortBin does a 16-way merge but this is configurable.
- Continue merging sorted blocks into bigger sorted blocks until only 1 block is left. That block contains the final sorted output.
Optionally, multiple threads may be used to speed up the sorting phase by running the quicksort algorithm on parallel CPU cores.
By default, the sort-then-merge strategy delegates disk I/O to a separate thread so that I/O and CPU processing can occur simultaneously.
## Performance
I compared the performance of SortBin to that of GNU sort.
This is in many ways an unfair comparison.
The GNU sort utility was designed to sort lines of text, is highly customizable and is expected to show reasonable performance on a wide range of computer systems.
In contrast, SortBin can only sort fixed-length records, is not customizable, and was optimized for sorting big files of small records on a modern PC.
I tested the performance by sorting a 100 GB file using only 1 GB memory.
For GNU sort, I allowed 2 GB memory because it seems to use a significant amount of memory to keep track of lines of text.
For SortBin, I set the number of parallel sort threads to 4.
I tested on Linux, x86\_64 CPU with 6 cores, 3600 MHz, 5980 rpm SATA harddisk.
My performance figures should be taken with a big pinch of salt.
I have not checked how accurate these results reproduce on the same system or on other systems.
And I did not try very hard to keep background processes from potentially influencing the results.
| Input size | Records | Record size | Program | Duration | CPU time |