Sort and deduplicate lines by a field

Bash

Sorts on the second column and keeps the first row of each group.

sort -t',' -k2,2 -u contacts.csv

# Count duplicates instead of removing them:
cut -d',' -f2 contacts.csv | sort | uniq -c | sort -rn

Download

Text Processing sort uniq fields chris

More in Text Processing

Pretty-print and query JSON Text Processing jq for the common cases: reformat, pick a path, filter an array, and reshape. Extract matches with a regular expression Text Processing grep -o prints only the matching part, which turns grep into a crude extractor. Replace text across a whole project Text Processing ripgrep finds the files, sed edits them in place. The dry run first is not optional. Rewrite a CSV column with perl Text Processing A one-liner that upper-cases the second field of every row, leaving the rest alone. Sum a column with awk Text Processing Adds the third field of every line and prints the total, ignoring anything non-numeric. A regular expression for ISO 8601 dates Text Processing Matches a date, optionally with a time and a zone. Deliberately strict about separators.

Random picks

Escape output in a PHP template PHP htmlspecialchars with the flags spelled out, because the defaults have changed between versions. Read a large file without loading it into memory Python Iterating a file object yields one line at a time, which is the difference between a constant footprint and an OOM. Throttle with a trailing call JavaScript Runs at most once per interval but still fires once more after the last event, so the final state is never dropped. Find the commit that introduced a bug Git Binary search across history. Mark a known good and a known bad commit and git narrows it down for you. Find slow queries in Postgres Databases Ranks statements by total time spent. Needs the pg_stat_statements extension enabled. Group rows into a JSON array Databases Returns one row per author with their posts nested, instead of a join that repeats the author on every row. Stop tracking a file without deleting it Git For the config file that should never have been committed. Add it to .gitignore afterwards. Table sizes, largest first Databases Where the disk actually went, including indexes and TOAST.