Delete Duplicate Image Files

Suppose you have 30 thousands image files in many directories. You want to delete duplicates.

Here's a Perl script that solves the problem.

How to Use It

perl del_dup.pl --help

To find dup files in a dir:
 perl del_dup.pl dirpath

To find dup files in several dir:
 perl del_dup.pl dirpath1 dirpath2 dirpath3 …

To delete dup files:
 perl del_dup.pl --delete dirpath
or
 perl del_dup.pl --delete dirpath1 dirpath2 …

When there are duplicate files, the first one found (in the order the dir is given) is preserved, the others are deleted.

To see this help again:
 perl del_dup.pl --help

Note: the options --help and --delete must be first argument.

A file is considered duplicate of another only if the 2 file's content are exactly identical. If you have 2 images, one is scaled version of the other, they are not considered identical.

I use this script on 30 thousand image files regularly over the years. For over 30k files, it runs under a minute.

Sample Output

perl del_dup.pl --delete C:\Users\h3\Pictures\keyboard "C:\Users\h3\Pictures\keyboard - Copy"

Adding dir to check: C:\Users\h3\Pictures\keyboard
Adding dir to check: C:\Users\h3\Pictures\keyboard - Copy
There are a total of 32 files examed.
==============================
There are 16 unique file size.
==============================
---------------------
These following files are identical:
C:\Users\h3\Pictures\keyboard/windowslogo.gif
C:\Users\h3\Pictures\keyboard - Copy/windowslogo.gif

C:\Users\h3\Pictures\keyboard/DSC_1108.jpg
C:\Users\h3\Pictures\keyboard - Copy/DSC_1108.jpg

C:\Users\h3\Pictures\keyboard/ms-sidewinder-x6-gaming-keyboard-full.jpg
C:\Users\h3\Pictures\keyboard - Copy/ms-sidewinder-x6-gaming-keyboard-full.jpg

C:\Users\h3\Pictures\keyboard/g510.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510.jpg

C:\Users\h3\Pictures\keyboard/71Uvd2tZOZL._AA1500_.jpg
C:\Users\h3\Pictures\keyboard - Copy/71Uvd2tZOZL._AA1500_.jpg

C:\Users\h3\Pictures\keyboard/g510 red.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510 red.jpg

C:\Users\h3\Pictures\keyboard/ms x4.jpg
C:\Users\h3\Pictures\keyboard - Copy/ms x4.jpg

C:\Users\h3\Pictures\keyboard/81fuOEG-2lL._AA1500_.jpg
C:\Users\h3\Pictures\keyboard - Copy/81fuOEG-2lL._AA1500_.jpg

C:\Users\h3\Pictures\keyboard/g110.jpg
C:\Users\h3\Pictures\keyboard - Copy/g110.jpg

C:\Users\h3\Pictures\keyboard - Copy/81hTgnd037L._AA1500_.jpg
C:\Users\h3\Pictures\keyboard/81hTgnd037L._AA1500_ - Copy.jpg
C:\Users\h3\Pictures\keyboard - Copy/81hTgnd037L._AA1500_ - Copy.jpg
C:\Users\h3\Pictures\keyboard/81hTgnd037L._AA1500_.jpg

C:\Users\h3\Pictures\keyboard/lenovo_thinkpad_usb_trackpoint_keyboard-2.jpg
C:\Users\h3\Pictures\keyboard - Copy/lenovo_thinkpad_usb_trackpoint_keyboard-2.jpg

C:\Users\h3\Pictures\keyboard/g19.jpg
C:\Users\h3\Pictures\keyboard - Copy/g19.jpg

C:\Users\h3\Pictures\keyboard/g510 yellow - Copy.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510 yellow - Copy.jpg

C:\Users\h3\Pictures\keyboard/g510 green.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510 green.jpg

==============================
There are 16 redundant files, totaling 2396674 bytes.
The following files (if any) will be deleted (if you used the “--delete” option):
C:\Users\h3\Pictures\keyboard - Copy/71Uvd2tZOZL._AA1500_.jpg
C:\Users\h3\Pictures\keyboard - Copy/81fuOEG-2lL._AA1500_.jpg
C:\Users\h3\Pictures\keyboard - Copy/81hTgnd037L._AA1500_ - Copy.jpg
C:\Users\h3\Pictures\keyboard - Copy/81hTgnd037L._AA1500_.jpg
C:\Users\h3\Pictures\keyboard - Copy/DSC_1108.jpg
C:\Users\h3\Pictures\keyboard - Copy/g110.jpg
C:\Users\h3\Pictures\keyboard - Copy/g19.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510 green.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510 red.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510 yellow - Copy.jpg
C:\Users\h3\Pictures\keyboard - Copy/g510.jpg
C:\Users\h3\Pictures\keyboard - Copy/lenovo_thinkpad_usb_trackpoint_keyboard-2.jpg
C:\Users\h3\Pictures\keyboard - Copy/ms x4.jpg
C:\Users\h3\Pictures\keyboard - Copy/ms-sidewinder-x6-gaming-keyboard-full.jpg
C:\Users\h3\Pictures\keyboard - Copy/windowslogo.gif
C:\Users\h3\Pictures\keyboard/81hTgnd037L._AA1500_.jpg
File deletion done (if any)!

Buy

Use the paypal button below, pay $5. In the comment field, put “perl delete dup”. I'll email you the program. Make sure your email address is included and correct.

Legal disclaimer: this software is sold as is. I'm not responsible for any damages caused by this software.