Skip to main content

Text processing

Convert file encoding

Get available encodings.

iconv -l

Convert text from the ISO 8859-15 character encoding to UTF-8.

iconv -f ISO-8859-15 -t UTF-8 < input.txt > output.txt

Get number of lines in targetFile

wc -l targetFile | grep -Eo '[0-9]+'

Read csv file and extract certain columns in certain order

awk -F ',' '{print $3 "," $1}' a1.csv > b2.csv
  • Saves output to file b2.csv.
  • Columns are referenced using $1, $2, $x.
  • Default fieldsep if not set is space.

Remove duplicate lines

Preserving line order in output

awk '!seen[$0]++' target.csv

Not maintaining line order

sort -u target.csv
  • -f - Case insensitive comparisons.


Delete lines matching regex & print result

sed '/regex/d' file

Replace all occurrences of regex in target file with string

sed 's/regex/string/g' file

Replace only on lines matching the line pattern

sed '/lineregex/s/regex/string/g' file
  • Remove g from any of the expressions above to replace only the first occurrence on each line.

Useful options

  • -i - Make changes overwriting the file.
  • --in-place=.bkp - Also update in-place but create a backup of the original file with .bkp extension.
  • -e - Apply multiple expressions (i.e. sed -e 's/regex0/string0/' -e 's/regex1/string1/' file).
  • -r - Allow extended regular expressions.

Sort by first column alphabetically, second numerically

sort -k1,1 -k2,2nr
  • Last r causes reversed output.
  • Default fieldsep is space.