Tool to deduplicate a list. Doubles or duplicates are a repeated redundant information present twice (or more) in a list, usually useless.
Duplicates in a List - dCode
Tag(s) : Data Processing
dCode is free and its tools are a valuable help in games, maths, geocaching, puzzles and problems to solve every day!
A suggestion ? a feedback ? a bug ? an idea ? Write to dCode!
A duplicate is a redundant element (double or triple or more) in a set of elements (a list).
Indicate / copy-paste items in the List field. dCode detects if the list is a list of terms (one per line) or a list of words (with separator) and finds redundant items repeated more than once in order to un-duplicate.
Example: A,B,C,D,A,B,C,A,B,A can be deduplicated as A,B,C,D (repeated terms were removed)
Be carefull to take into account all writing variants of a same element: ignore accents and diacritics allow to treat words like item and ìtém as a same word twice. Also, ignoring uppercase and lowercase allow to detect/find item and ITEM as a duplication of the same word and delete them.
By listing items, dCode detector counts the number of appearance of each, the list can be sorted to detect the most duplicated ones.
dCode counts duplicates/redundancies, so 2 identical items is counted 1 duplicate. N identical elements is counted N-1.
There are many applications where the presence of duplicates can be a problem:
— detect and delete duplicate files to free up disk space
— deduplicate data from a database, retaining only one occurrence of each unique/distinct record.
— sending mailings by checking if a list of email addresses contains duplicates in order to filter the list and delete identical emails.
dCode retains ownership of the "Duplicates in a List" source code. Except explicit open source licence (indicated Creative Commons / free), the "Duplicates in a List" algorithm, the applet or snippet (converter, solver, encryption / decryption, encoding / decoding, ciphering / deciphering, breaker, translator), or the "Duplicates in a List" functions (calculate, convert, solve, decrypt / encrypt, decipher / cipher, decode / encode, translate) written in any informatic language (Python, Java, PHP, C#, Javascript, Matlab, etc.) and all data download, script, or API access for "Duplicates in a List" are not public, same for offline use on PC, mobile, tablet, iPhone or Android app!
Reminder : dCode is free to use.
The copy-paste of the page "Duplicates in a List" or any of its results, is allowed (even for commercial purposes) as long as you credit dCode!
Exporting results as a .csv or .txt file is free by clicking on the export icon
Cite as source (bibliography):
Duplicates in a List on dCode.fr [online website], retrieved on 2024-08-26,