How to Remove Duplicates from CSV?

Software & Web Applications
1

Can anyone suggest to me the best solution to remove duplicates from CSV?

2

Hi,
I’m also doing this every week at work.
Being an old Unix nerd, I usually do this using something like:

cat file.csv | sort | uniq > new_file.csv

The only downside is that the resulting file is sorted but that’s mostly ok for my use cases.

I also have a small python script I’m using when I need to count the number of duplicates. Another use case, but if you need it, I can paste it in another reply.

I hope it helps.
/Johan

3