DE EN

• About AllDup
• Product Description
• System Requirements
• Important Notes

• AllDup Main Window
• Main Window
• Source Folder
• Search Methods
• Search Options
• Archive Files
• Database
• File Filter
• Folder Filter
• Windows Environment Variables
• File Preview
• Profiles
• Saved Search Results
• Options
• Update Tool

• Search Result
• Search Result
• Menu Bar / Toolbar
• Context Menus
• File Manager
• Keyboard Shortcuts

• Miscellaneous
• Command Line
• Text Placeholders
• Date and Time Placeholders
• Speed Comparison Byte-by-Byte vs. SHA-1

Speed comparison Byte-for-byte vs. SHA-1

Here you can find information about a speed comparison between the search methods Byte-for-byte and SHA-1. For the speed comparison, a computer with Windows 7 x64, 8 GB RAM and an i3 CPU was used. The test data is stored on an external USB2 hard drive, which is encrypted with the TrueCrypt software. Furthermore, for the test, the options Ignore Exif data of JPEG files and Log search statistics of the progress indicator every x minutes were activated.

Test with 'Byte-for-byte':

13:29:36 - Search: File content Byte-for-byte 13:29:37 - File count: 45442 13:29:37 - Searching: R:\\!P2 13:33:37 - Duration : 0:04:01, Progress : 7 %, Current File : 3361 \\ 45442, Duplicates found : 1377 (3 %) (2.83 GB), Groups : 584, File comparisons : 811 13:37:39 - Duration : 0:08:03, Progress : 16 %, Current File : 7652 \\ 45442, Duplicates found : 2717 (5 %) (5.58 GB), Groups : 999, File comparisons : 1776 13:41:39 - Duration : 0:12:03, Progress : 45 %, Current File : 20806 \\ 45442, Duplicates found : 14426 (31 %) (9.30 GB), Groups : 3678, File comparisons : 11545 13:45:39 - Duration : 0:16:03, Progress : 57 %, Current File : 26252 \\ 45442, Duplicates found : 20138 (44 %) (11.94 GB), Groups : 4462, File comparisons : 16705 13:49:39 - Duration : 0:20:03, Progress : 65 %, Current File : 29665 \\ 45442, Duplicates found : 23521 (51 %) (14.16 GB), Groups : 4776, File comparisons : 19873 13:53:39 - Duration : 0:24:03, Progress : 70 %, Current File : 32144 \\ 45442, Duplicates found : 26118 (57 %) (16.41 GB), Groups : 5015, File comparisons : 22286 13:57:39 - Duration : 0:28:03, Progress : 76 %, Current File : 34663 \\ 45442, Duplicates found : 28575 (62 %) (18.42 GB), Groups : 5084, File comparisons : 24733 14:01:39 - Duration : 0:32:03, Progress : 82 %, Current File : 37573 \\ 45442, Duplicates found : 30751 (67 %) (20.45 GB), Groups : 5267, File comparisons : 26992 14:04:51 - 32215 duplicates with 22,970,559,143 bytes found in source folder 'R:\\!P2' 14:04:51 - Groups: 5,442 14:04:51 - File comparisons: 28,370 14:04:51 - Duplicates: 32215 (70%) (21.39 GB) 14:04:51 - Time spent: 00:35:15

Test with 'SHA-1':

14:10:31 - Search: File content SHA-1 (160-bit) 14:10:32 - File count: 45442 14:10:32 - Searching: R:\\!P2 14:14:32 - Duration : 0:04:01, Progress : 7 %, Current File : 3422 \\ 45442, Duplicates found : 1487 (3 %) (3.08 GB), Groups : 639, File comparisons : 867 14:18:32 - Duration : 0:08:01, Progress : 26 %, Current File : 12073 \\ 45442, Duplicates found : 5778 (12 %) (6.98 GB), Groups : 2400, File comparisons : 3689 14:22:32 - Duration : 0:12:01, Progress : 55 %, Current File : 25020 \\ 45442, Duplicates found : 18904 (41 %) (11.16 GB), Groups : 4373, File comparisons : 15542 14:26:32 - Duration : 0:16:01, Progress : 66 %, Current File : 30193 \\ 45442, Duplicates found : 24082 (52 %) (15.22 GB), Groups : 4816, File comparisons : 20396 14:30:32 - Duration : 0:20:01, Progress : 78 %, Current File : 35462 \\ 45442, Duplicates found : 29392 (64 %) (19.56 GB), Groups : 5116, File comparisons : 25525 14:34:12 - 32215 duplicates with 22,970,559,143 bytes found in source folder 'R:\\!P2' 14:34:12 - Groups: 5,442 14:34:12 - File comparisons: 28,370 14:34:12 - Duplicates: 32215 (70%) (21.39 GB) 14:34:12 - Time spent: 00:23:41

Conclusion
As expected, the search method SHA-1 is faster than Byte-for-byte. Comparing checksums stored in RAM significantly reduces file access to the hard drive, which can save a lot of time.