How to find/identify identical duplicate files (Windows 10)?

B0bbyEwing

Before you don't read the issue & instantly respond with something like "dupeguru" then please read…

I've used a couple duplicate file finder programs in the past & they've found things. Now whether they've found everything I'm beginning to question because I accessed a drive that I'd recently ran one of these pieces of software on & I found that there was at least 1 file that was duplicated which hadn't been picked up. I ran another piece of software on just this folder alone & this also didn't detect any duplicates apparently. I can assure that the file is the same though.

Same - file size, run time (video file), bitrate, so on & so forth (when looking at the right hand pane within Windows 10).

The only differences are the file name & the downloaded date.

I know they're the same because I downloaded from the exact same source, just like a month or so apart as I'd forgotten I'd downloaded it & just named it something different on the second time I downloaded it.

So it's not like I downloaded from one website on one date & the same video file from elsewhere on another date where perhaps one is in 720p & the other is in 1080p & you could maybe pass this off as the reason why (but then the file size would be different in that case & the dimensions etc so that wouldn't apply). I'm talking about downloading it from the exact same website.

In a way that a beginner would understand - how can I check whether their makeup is the same or not? Even though I know it's 100% the same content within.

As my concern now, as I have many TBs to run through (& de-duplicate), is that this software is going to miss an absolute ton of files & instead of freeing up say 8TB I'll only free up 1TB or something.

booneruk

There's a whole computer science discussion that can be had here, but files that look to be the same (to a human) can be different at a byte level. For example, a text file that looks the same to a human eye can be entirely different at byte level if it's encoded with UTF8 or ASCII.

The same file downloaded multiple times can be subtly different too, if a video or image has gone through a CDN it can be encoded differently depending on the device requesting it, the jpeg compression level could be subtly different for example - no difference to a casual human observer but different at a byte level. There could also be differences in the meta data between two downloads of the same file.

I've had a quick read of dupeguru's documentation and there appears to be a picture/audio mode which may help with those types. I'm not sure about video, it could be much more of a problem for computer logic to determine two video files with byte level differences are essentially the same).

It might be very hard to achieve what you want here I'm afraid. Maybe there are other tools or an AI that can help.

Newcad

The first thing to note is that any duplicate finder will only find what you ask it to find.
ie. if you tell it to compare names then two otherwise identical files with different names will not be found.
If you tell it to compare dates then two otherwise identical files with different dates will not be found.
And so on, and the more comparison fields you set then more they can fail to find duplicates.

The best way is to compare the contents of the files, ignoring names and dates.
The fastest way to do that is by comparing each file's checksum.

Then you have the issue of slight differences, eg. a photo file that has had say red-eye reduction applied to it is not identical to the original file, the checksum of each one will be different.
But are they still close enough to be called duplicates? It depends on your point of view.

That is why most duplicate image finders let you set how close a match you want images to be.
However I will note that most of the ones I've tried will find one set of close matches when first run, but if you delete one set of the near duplicates and run the finder again with the exact same settings it will find different close matches.
By giving it some leeway to find closely similar but not exactly identical files then you give it the chance to do find different similarities each time it is run.

(Unfortunately the very fast 'exact' duplicate finder that I use myself and that I used to recommend is no longer generally available. It's part of a particular application which has been updated and the new duplicate finder now included is slower. I also consider the new one to be too dangerous for the unwary. I've seen people need to reinstall Windows after running it without being careful and deleting duplicate System and Application files, so I no longer recommend it).

About all you can do is find one (or two) finders that you like and stick with them.
Personally I use that fast checksum checker to find exact duplicates, then for images I'll use a much slower 'similar image' finder such as VisiPics.

mta999

If it were me what I would do is make a copy of file 1 and then compare the two files and see if it shows them as the same

if it does then you know it is a file difference with file 2 and if it doesn't you know it is a program problem

B0bbyEwing

booner - I think I kind of get what you're saying.

But with your reply I still don't get why downloading from the same source the file could be different.

Ok now I didn't download the video from imgur but this is basically how I can explain my situation…

So for example, say I hopped on imgur & found some random video file: https://imgur.com/gallery/right-man-right-place-right-time-zal3qAq I've also left the URL as is for those who don't like it to be put within text & may think it's nefarious :)

I download it. Call it video 1.

Then in a months time I hop on imgur, scroll around, come across that exact same URL, that exact same video & I save it again as I forgot I saved it last time. I call it video 1a.

Now at my level of understanding, that's the exact same video file, just named differently & on different dates. I can't grasp how that file may be different the second time round.

Newcad - when using Auslogics (my go to) in the beginning, many months ago I ran numerous tests before I used it because it wasn't finding duplicates when I knew there were some. I would take a picture file, copy it but rename it & have them within a folder & then run Auslogics on this folder with various settings to see which settings would detect the duplicate. Once it flagged correctly, that's the settings I then used going forward.

So the setting was irrespective of file name or date.

And it's with those settings in place that it's not flagged duplicates this time round.

booneruk

But with your reply I still don't get why downloading from the same source the file could be different.

A lot of media will be delivered through CDNs on the web - these optimise the content for the device requesting so a video you request in a browser window that's full screen could be delivered in higher quality than the same video requested in a smaller window. There also might also be unique tracking / copyright data embedded in the files.

These are just examples, but essentially it's very plausible that there could be subtle differences in the same file downloaded twice that would cause a tool to identify the files as different

B0bbyEwing

https://forums.moneysavingexpert.com/discussion/comment/82043648#Comment_82043648

Well that's going to be an absolute ballache when it comes to freeing up space on my drives then.

Overall there'll be 100s of thousands.

I'll just have to let the software do its thing & that'll be the best of a bad job then I guess.

Is there any duplicate software that's generally seen as better than the rest? As I say, I pretty much use Auslogics but that's only because I'm comfortable with it.

The free version.

I've no idea if this is way down the ladder or not in terms of quality.

debitcardmayhem

Question when you downloaded copy One did you check its checksum with the source and ditto copy Two . That is just one possible reason for the difference i.e. corruption doing downloads, 1 bit makes no visible difference necessarily but if it is a program it could be a big way it screws you. IMHO the best way to compare files is to use checksums , not foolproof but the best chance of spotting differences Iwrote my own scripts to do this years ago, but there are ,I am sure, apps(programs…of course) to do this too.

Sadly folks want easy to look after their lax ways🤣

B0bbyEwing

https://forums.moneysavingexpert.com/discussion/comment/82043981#Comment_82043981

I wouldn't know how to check the original tbh.

And no I haven't checked the makeup of file 1 & 2. Again I wouldn't know how.

I read about Powershell & tried it but clearly I don't have a clue what I'm doing because it did absolutely nothing.

Just to spin off your last comment there - some folk maybe want it easy because if you don't have a clue what you're doing (points at self here) & you don't 'get it' in explanation 1 then some people can get quite shirty.

Though I've only found this to be the case online & not in person. In person people tend to be more patient & understand everyone learns at different paces. Online though it's like that screen gives them some kind of pass to be a <insert whatever name you want to call it>.

debitcardmayhem

https://forums.moneysavingexpert.com/discussion/comment/82044122#Comment_82044122

Fyi a little background , this might explain a little using powershell https://www.geeksforgeeks.org/techtips/verify-checksum-of-a-file-in-windows/

or in command prompt use certutil https://learn.microsoft.com/en-us/windows-server/administration/windows-commands/certutil

I was a little distracted earlier but I have 2 mins now so hope these help😎

B0bbyEwing

https://forums.moneysavingexpert.com/discussion/comment/82044214#Comment_82044214

Hey, careful you don't get accused of using nefarious links!

Thanks for that. I was given the Powershell command but seems I was only given half the step. Your link explains much better how to use it properly so I'll do that.