Useful CLI commands

These are mostly Debian GNU/Linux command line interface commands.

Commands

Linux kernel stuff

Flush cache.

$ sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"

ansi2txt

Part of the colorized-logs Debian package.

Remove color codes from text.

 $ ansi2txt < ./input.txt > ./output.txt

apt

Check which packages depend on package package-name.

$ apt rdepends package-name

Install upgrades without prompts

$ sudo DEBIAN_FRONTEND=noninteractive apt upgrade -y

base64

Calculate MD5 checksum of a binary file.

$ openssl md5 -binary foo.txt | base64  # with openssl and base64
$ md5sum --binary foo.txt | cut -d' ' -f1 | xxd -r -p | base64  # with md5sum, xxd and base64

bash

Manual.

Check if $stringA comes alphabetically before $stringB using current locale.

$ if [[ "$stringA" < "$stringB" ]]; then echo true; else echo false; fi;

Search find results with grep using the exit code from a statement evaluated by the built-in if.

$ mkdir a;
$ touch ./a/foo.txt
$ touch ./a/bar.txt
$ if results="$(find . -type f | grep 'foo')"; then printf "Found:\n%s\n" "$results"; else printf "Found nothing.\n"; fi;
Found:
./a/foo.txt
$ if results="$(find . -type f | grep 'baz')"; then printf "Found:\n%s\n" "$results"; else printf "Found nothing.\n"; fi;
Found nothing.
$ if results="$(find . -type f | grep '.txt$')"; then printf "Found:\n%s\n" "$results"; else printf "Found nothing.\n"; fi;
Found:
./a/bar.txt
./a/foo.txt

Use modulo integer math to print the integers between 0 and 100 that are divisible by 5.

#!/bin/bash
for ((i=0; i<=100; i++)); do
  if [ $((i % 5)) -eq 0 ]; then
    echo $i
  fi
done

Count the number of occurrences of a single ASCII character (e.g. ,) (see ref)

#!/bin/bash
var="text,text,text,text"
res="${var//[^,]}"
echo "$res"
echo "${#res}"

Quotation rules

How to insert a single apostrophe into an already single-apostrophe-quoted string.

$ echo "You can't do it easily" | sed -E -e 's/can'\t/can/'  # sed example
You can do it easily.

Bash input/output

Provide string to a command via stdin. The following are equivalent.

$ echo "666" | sed 's/6/7/g';              # Uses pipe format.
777
$ myVar="666"; sed 's/6/7/g <<< "$myVar";  # Uses “here string” format.
777

Supply the first line of a text file as an argument.

$ fpass=/dev/shm/password.txt;           # create file
$ touch "$fpass" && chmod 700 "$fpass";  # set to user-only permission
$ echo "hunter2" > "$fpass";             # save password to file
$ echo "$(head -n1 "$fpass")";     # read contents of file as argument

Bash job management

See https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Job-Control

Push job into background.

^Z   # (i.e. Ctrl-Z)
[1]+ Stopped     git tag --sign 2023-04-15 -m "SignStamp state" 6e31b1a

Get a list of current background jobs

$ jobs
[1]+ Stopped     git tag --sign 2023-04-15 -m "SignStamp state" 6e31b1a

Restart a job (e.g. one with jobspec 1) in the background and check that it is running.

$ bg 1
[1]+ git tag --sign 2023-04-15 -m "SignStamp state" 6e31b1a &
$ jobs
[1]+ Running     git tag --sign 2023-04-15 -m "SignStamp state" 6e31b1a &

Detach a job (e.g. one with jobspec 1) from the current shell to permit safe exit without terminating the job.

$ disown %1

Bash string manipulation

See Manipulating Strings.

Replace substring in variable.

$ myVar="foobarbaz";
$ echo "${myVar/bar/qux}";
fooquxbaz

Get first 7 characters of a string.

$ myVar="deadbeef";
$ echo "${myVar:0:7}"
deadbee

Get last 4 characters of a string.

$ myVar="deadbeef";
$ echo "${myVar:(-4)}";
beef

Lowercase characters in a variable

$ myVar="fooBARbaz";
$ echo "${myVar,,}";
foobarbaz

bc

Evaluate math expressions

$ echo "scale=12; 1 / 1.61803398875" | bc -l  # division with 12 decimal places
.618033988749
$ echo "scale=6; l(2.718281)" | bc -l  # logarithm with only 6 decimal places
.999999

brew

Homebrew is a package manager for macOS.

Install via script and curl (simple, but sketchy method available at https://brew.sh ):

$ /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"

Update packages

$ brew update

Get info on a package

$ brew info coreutils

Install a package

$ brew install coreutils

Test package commands

$ gdate -Is  # Equivalent to Debian 12 `$ date -Is`.
2024-10-08T20:03:18+00:00

chmod

References:

• https://www.pluralsight.com/blog/it-ops/linux-file-permissions
• https://linuxconfig.org/chmod

Remove read, write, and execute permissions for others to a file or directory named foo.

# chmod o-rwx foo

Add read and execute permissions for group to a file or directory named foo.

# chmod g+rx foo

Make all directories openable (i.e. executable) by user starting at a directory named bar.

# find ./bar -type d -exec chmod u+x \;

Make all files non-executable by everyone within a directory named bar.

# find ./bar -type f -exec chmod ugo-x \;
# find ./bar -type f -exec chmod -x \;

Make a file named foo executable for user, group, and others (i.e. everyone).

# chown +x foo

choom

Get OOM score for process with PID 1

$ choom --pid 1

List the bottom 50 processes in the OOM priority list (during low memory, the bottom is killed first).

$ printf "pid,ooms,name\n"; while read -r line; do pid="$(echo "$line" | cut -d' ' -f2)"; name="$(echo "$line" | cut -d' ' -f11-)"; ooms="$(choom -p "$pid" | grep -Eo "[0-9]+$" | head -n1)"; printf "%9d,%4d,%s\n" "$pid" "$ooms" "$name"; done < <(ps aux | tr -s ' ' | tail -n+2) | sort -k2 | tail -n50

chown

Change ownership of a file or directory named baz to user debuser and group debuser.

# chown debuser:debuser baz

Change ownership of all files and directories contained within the directory named foo to user debuser and group debuser.

root@h: chown -R debuser:debuser foo

chrony

The following commands assume the package chrony is installed.

Get a list of NTP refclocks being used to adjust local time:

$ chronyc sources

Get current machine's statistics (e.g. get Frequency or how slow or fast local clock is)

$ chronyc tracking

convert

See #ImageMagick

crontab

Edit crontab.

$ crontab -e

Print crontab to stdout:

$ crontab -l

Erase crontab (DON'T DO UNLESS YOU HAVE A BACKUP OF THE CRONTAB):

$ crontab -r

Print to stdout the crontab of user www-data via user root (ref):

$ sudo su - www-data -s /bin/bash -c "crontab -l"

cryptsetup

Get details on a volume encrypted with LUKS

$ cryptsetup status /dev/mapper/$some_volume

where $some_volume is the name of an encrypted volume. Reference.

date

Note: Assumes GNU date.

Print today's calendar date in ISO 8601 format

$ date -I
$ date -Id
$ date +%Y-%m-%d

Print current date and time in ISO 8601 format to second resolution.

$ date -Is                    # e.g. 2024-02-01T05:44:58+00:00
$ date +%Y-%m-%dT%H:%M:%S%:z  # e.g. 2024-02-01T05:44:58+00:00

Print current date and time in ISO 8601 format to second resolution with minimal separators. (e.g. for file name use)

$ date +%Y%m%dT%H%M%S%z  # e.g. 20240201T054458+00

Print current UNIX epoch. (i.e. integer seconds since 1970-01-01.)

$ date +%s

Print all calendar dates for the next 365 days. (e.g. 2024-02-01\n2024-02-02\n2024-02-03\n…)

#!/bin/bash
today="$(date +%s)"; n=0; for dia in {0..365}; do
  day="$((today + dia * (24*60*60) ))";
  date --date="@$day" "+%Y-%m-%d";
done;

dd

Read every block of a block device /dev/sdb in 4 KiB increments.

$ sudo dd if=/dev/sdb of=/dev/random bs=4k

Create a 10 MiB file containing pseudorandom noise using 2 MiB of RAM at a time.

$ dd if=/dev/urandom of=sample.txt bs=2M count=5

dig

Get public IP address:

$ dig +short myip.opendns.com @resolver1.opendns.com
$ dig +short txt ch whoami.cloudflare @1.0.0.1
$ dig TXT +short o-o.myaddr.l.google.com @ns1.google.com

dpkg

List available kernels.

$ dpkg --list | grep -- linux-image

Check which package owns a file.

$ dpkg -S /etc/systemd/logind.conf  # by file path
$ dpkg -S "$(which zdump)"          # by command name, e.g. `zdump`

dpkg-reconfigure

Add a locale in Debian-based systems that use dpkg.

$ sudo dpkg-reconfigure locales

Navigate menus to select the local. Recommended: locales ending in .UTF-8 (e.g. C.UTF-8 (compatibility), en_US.UTF-8 (English of United States), zh_CN.UTF-8 (Chinese of mainland China), id_ID.UTF-8 (Indonesian of Indonesia), ja_JP.UTF-8 (Japanese of Japan), ko_KR.UTF-8 (Korean of South Korea).

dstat

Consider using dool (GitHub)

Show system stats, averaged every 60 seconds per line

 dstat --time --load --proc --cpu --mem --disk --io --net --sys --vm 60

emacs

See Emacs notes

exiftool

Note: Consider using BK-2020-03bkphotorights script to add XMP data with Creative Commons attribution data.

List all EXIF data, including XMP tags.

$ exiftool file.jpg

Remove all EXIF data from photograph files

$ exiftool -all= file.jpg

Remove only GPS EXIF data from JPG (see https://exiftool.org/forum/index.php?topic=6037.0 )

$ exiftool -gps:all= file.jpg

remove only GPS EXIF data from JPG If GPS is in XMP:

$ exiftool "-gps*=" file.jpg

Rotate image via EXIF tag:

$ exiftool -Orientation#=1 file.jpg   # Horizontal (normal)
$ exiftool -Orientation#=2 file.jpg   # Mirror horizontal
$ exiftool -Orientation#=3 file.jpg   # Rotate 180
$ exiftool -Orientation#=4 file.jpg   # Mirror vertical
$ exiftool -Orientation#=5 file.jpg   # Mirror horizontal and rotate 270 CW
$ exiftool -Orientation#=6 file.jpg   # Rotate 90 CW
$ exiftool -Orientation#=7 file.jpg   # Mirror horizontal and rotate 90 CW
$ exiftool -Orientation#=8 file.jpg   # Rotate 270 CW

f3

Fight Fake Flash (F3) is a utility for detecting fake flash storage drives.

Install

$ sudo apt install f3

Run on drive mounted at /media/baltakatei/myusb

$ f3write /media/baltakatei/myusb

Verify

$ f3read /media/baltakatei/myusb

ffmpeg

Encode h264 video for compatibility with Firefox.

$ ffmpeg -i input.mp4 -c:v libx264 -pix_fmt yuv420p output.mp4
$ ffmpeg -i input.mp4 -c:v libx264 -preset veryslow -crf 18 -pix_fmt yuv420p output.mp4  # higher quality
$ ffmpeg -i input.mp4 -c:v libx264 -preset veryslow -crf 26 -pix_fmt yuv420p output.mp4  # smaller size

Extract clip with time codes.

$ ffmpeg -i input.mp4 -ss 00:01:00.000 -to 00:03:00.000 -avoid_negative_ts 1 output.mp4
$ ffmpeg -i input.mp4 -ss 00:01:00.000 -to 00:03:00.000 -c:v libx264 -preset slow -crf 18 -c:a aac -b:a 128k -avoid_negative_ts 1 output.mp4      
$ ffmpeg -i input.mp4 -ss 60 -t 120 -avoid_negative_ts 1 output.mp4

Extract frame of video to save as PNG file (e.g. the first frame).

$ ffmpeg -i input.webm -ss 00:00:00 -frames:v 1 output.png

Extract album artwork from one of the original audiobook mp3 files. (see Audiobook transcoding notes.)

$ ffmpeg -i How\ To\ -\ Track\ 001.mp3 -an -vcodec copy album_artwork.png

Quality of life

Hide verbose configuration banner. (e.g. `built with gcc 11… configuration: --prefix=/usr…`)

$ ffmpeg -hide_banner

Encode video

Encode video using VP9 codec with 2 passes and tile-based multithreading.

#!/bin/bash
fin=input.mkv;
ffmpeg -nostdin -i "$fin" -c:v libvpx-vp9 -row-mt 1 -b:v 0 -crf 18 -pass 1 -f null /dev/null && \
ffmpeg -nostdin -i "$fin" -c:v libvpx-vp9 -row-mt 1 -b:v 0 -crf 18 -pass 2 "${fin%.mkv}.webm"

Specify a constant rate factor (CRF)

$ ffmpeg -i input.mp4 -c:v libx264 -preset slow -crf 18 -c:a copy output.mp4

Clip a video at CRF 18 and encode audio to OPUS.

$ ffmpeg -i input.mp4 -ss 00:01:00.000 -to 00:03:00.000 -c:v libx264 -preset slow -crf 18 -c:a libopus -b:a 128k -avoid_negative_ts 1 output.mp4

Process multiple videos in a Bash while loop. (Avoid stdin conflict with -nostdin option)

while IFS= read -r file; do
  ffmpeg -nostdin -i "$file" -c:v libx264 -c:a aac "${file%.avi}".mkv
done < <(find . -name '*.avi')

Apply video filters

Create a 1-minute 60 fps time lapse video from 3 hours of 30 fps input.

From videos, create a file list then run ffmpeg on the list.

$ for f in ./*.MP4; do echo "file '$PWD/$f'" >> filelist.txt; done

Create the 60 fps (-r 60) timelapse of 3 hours reduced into 1 minute. The setpts factor is equal to (duration out)/(duration in)*(fps out)/(fps in). So, (1*60)/(3*60*60)*(60/30) ≈ 0.01111)

$ ffmpeg -nostdin -f concat -safe 0 -i filelist.txt -vf "setpts=0.01111*PTS" -an -r 60 output_timelapse.mp4

Apply a temporal median filter across a radius of 10 frames.

$ ffmpeg -f -i input.mp4 -vf "tmedian=radius=10:planes=15:percentile=0.5" -an output.mp4

Apply both a time lapse and a temporal median filter for several .MP4 files.

$ for f in ./*.MP4; do echo "file '$PWD/$f'" >> filelist.txt; done;
$ ffmpeg -f concat -safe 0 -i filelist.txt -vf "setpts=0.01111*PTS, tmedian=radius=10:planes=15:percentile=0.5" -an -r 60 -crf 30 output_timelapse_crf30_median.mp4

Split a video file into roughly equal segments

Reference: https://unix.stackexchange.com/a/212518/411854

$ ffmpeg -i input.mp4 -c copy -map 0 -segment_time 00:20:00 -f segment -reset_timestamps 1 output%03d.mp4

Combine video files into a single file

Reference: https://trac.ffmpeg.org/wiki/Concatenate

# this is a comment of the file named mylist.txt
file '/path/to/file1.wav'
file '/path/to/file2.wav'
file '/path/to/file3.wav'

ffmpeg -f concat -safe 0 -i mylist.txt -c copy output.wav

Add subtitles to a video file

Add multiple ASS subtitle files to a single MP4 video file.

ffmpeg -i input.mp4 \
       -sub_charenc 'UTF-8' -f ass -i input.en-US.ass \
       -sub_charenc 'UTF-8' -f ass -i input.es-US.ass \
       -sub_charenc 'UTF-8' -f ass -i input.id.ass \
       -sub_charenc 'UTF-8' -f ass -i input.ja.ass \
       -map 0:v -map 0:a \
       -map 1 -map 2 -map 3 -map 4 \
       -metadata:s:s:0 language=eng \
       -metadata:s:s:1 language=spa \
       -metadata:s:s:2 language=ind \
       -metadata:s:s:3 language=jpn \
       -c copy \
       -c:s ass output.mkv

ffprobe

Get audio duration in seconds as a decimal number.

$ ffprobe -i "$file" -show_entries format=duration -v quiet -of csv="p=0";

Get audio duration in seconds of all .flac files in the working directory with bc.

#!/bin/bash
dur="0.0"; while read -r line; do
  line_dur="$(ffprobe -i "$line" -show_entries format=duration -v quiet -of csv="p=0")";
  dur="$(echo "$dur + $line_dur" | bc -l)";
done < <(find . -type f -name "*.flac");
echo "$dur";

Get chapter times.

$ ffprobe -i "$file" -print_format json -show_chapters -sexagesimal

find

Outputs newline-delimited (default) list of paths of files or directories matching specified filters.

List all files recursively

$ find . -type f

List all directories recursively

$ find . -type d

Search $HOME for files ending in .jpg or .JPG.

$ find $HOME/ -type f -iname "*.jpg"

Get file sizes, via du, of all files recursively

$ find . -type f -exec du -b '{}' \;

List all files and directories within one level of subdirectories

$ find . -maxdepth 1

List all files and directories precisely 4 subdirectories deep

$ find . -mindepth 4 -maxdepth 4

List all files recursively, following symlinks up to a maximum depth of 10 subdirectories deep

$ find -L . -maxdepth 10 -type f

List all files and directories starting with a . (i.e. list all dotfiles and dotdirs)

$ find . -name ".*"

List all files and directories starting with a .

$ find . ! -name ".*"

List all files recursively except those ending in either .ots or .ots.bak

$ find . -type f ! \( -name "*.ots" -or -name "*.ots.bak" \)

Find files of a minimum size. (e.g. greater than but not equal to 1 MiB)

$ find . -type f -size +$((1024 * 1024))c  # calc MiB
$ find . -type f -size +$((1024 ** 2))c    # calc MiB
$ find . -type f -size +1048576c           # use bytes
$ find . -type f -size +1M                 # do not use due to rounding issues

Find files of a maximum size. (e.g. less than but not equal to 1 MiB)

$ find . -type f -size -1048576c  # use -1048576c instead of -1M due to rounding issues

Find files older than a certain date (e.g. 2024-01-01)

$ find /path/to/search -not -newermt 2024-01-01

List files sorted by modification date

$ find /path/to/directory -type f -printf '%T@ %p\n' | sort -n | cut -d' ' -f2-

gcc

The GNU C Compiler

Compile C-code specified in the command line.

$ gcc -x c <(echo 'int main(){return 0;}')
$ echo 'int main(){return 0;}' | gcc -x c -
$ gcc -x c -o hello <(echo -e "#include <stdio.h>\n main()\n{\nprintf(\"hello, \");\nprintf(\"world\");\nprintf(\"\\\n\");\n}")

git

See recent checkout, commit, or other operations by commit reference. (e.g. to find a commit lost because it was on a detached HEAD).

$ git reflog

To automatically sign merges (not default).

$ git config merge.gpgsign true
$ git -c merge.gpgsign='true' pull

See remotes.

$ git remote -v

Rename a remote. (ref)

$ git remote rename beanstalk origin

Export a git bundle (repository backup)

$ git bundle create filename.bundle --all

Trace git operations (especially those involving gpg signing operations). (ref)

$ GIT_TRACE=1 git commit

Get current commit, short git log entry, and ISO-8601 date

$ git log -1 --pretty=format:"%h %s %cd" --date=iso
$ git log -1 --pretty=format:"%h %s %cd" --date=format:"%Y-%m-%d"  #shorter date

Delete a local branch named `develop` (assuming `develop` is not checked out).

$ git branch -d develop

Untrack but don't remove committed file. (Ref/attrib; useful if you tracked something that shouldn't be tracked like repo.git/config)

$ git rm --cached path/to/committed/file

Pull branch develop from remote origin to unchecked out local branch develop without altering working tree (useful if worktree files are being used by something else).

$ git pull origin develop:develop

Set single configuration parameters for the duration of a single command.

Disable checking GPG signatures when running $ git log. (see ref)

$ git -c log.showSignature='false' log

Commit and/or tag with ots --wait option.

$ pathOtsWait="/home/debuser/.local/share/ots/ots-git-gpg-wrapper-wait.sh";
$ git -c gpg.program="$pathOtsWait" commit -S
$ git -c gpg.program="$pathOtsWait" tag --sign "some_tag_name" main

Set a remote branch `origin/develop` as the upstream branch for a local branch named `develop`.

$ git branch --set-upstream-to=origin/develop develop

Get current commit, short git log entry, and ISO-8601 date

$ git log -1 --pretty=format:"%h %s %cd" --date=iso
$ git log -1 --pretty=format:"%h %s %cd" --date=format:"%Y-%m-%d"  #shorter date

Delete a local branch named `develop` (assuming `develop` is not checked out).

$ git branch -d develop

Untrack but don't remove committed file. (Ref/attrib; useful if you tracked something that shouldn't be tracked like repo.git/config)

$ git rm --cached path/to/committed/file

Pull branch develop from remote origin to unchecked out local branch develop without altering working tree (useful if worktree files are being used by something else).

$ git pull origin develop:develop

Set a remote branch `origin/develop` as the upstream branch for a local branch named `develop`.

$ git branch --set-upstream-to=origin/develop develop

Disable git credential helper for a single command.

$ git -c credential.helper= pull origin refs/heads/master

gpg

See also GnuPG

Use a temporary keyring

$ gpg --no-default-keyring --keyring /dev/shm/temp-keyring.kbx

Refresh keys

$ gpg --keyserver keys.openpgp.org --receive-keys deadbeef deadbeef
$ gpg --keyserver keyserver.ubuntu.com --receive-keys deadbeef deadbeef

grep

Search for a process named “bash” with ps aux and grep bash but exclude matches of “grep bash" itself.

$ ps aux | grep "bas[h]"

Ignore binary matches with -I option.

$ grep "a" largeProgram.exe && echo "Match found." || echo "No match found."
grep: get_put_char: binary file matches
Match found.
$ grep -I "a" largeProgram.exe && echo "Match found." || echo "No match found."
No match found.

Use find with parallel to recursively search a file tree for text matches.

$ find . -type f -name "*.tsv" | parallel grep -iHIC3 --color=always -e 'mexico' '{}'

-name "*.tsv": Search only files with names ending in .tsv.

-i: Ignore character capitalization.

-H: Print name of file containing match.

-C3: Show 3 lines before and after match.

-I: Do not search binary files.

-e 'mexico': Search for lines containing the string mexico

'{}': Filename word placeholder for parallel.

gs

See also Ghostscript.

Remove raster images from a PDF.

 $ gs -o noimages.pdf -sDEVICE=pdfwrite -dFILTERIMAGE input.pdf

Regular Expressions

Show lines that match pattern.

$ grep 'some pattern' -- file.txt

Show lines that don't match pattern.

$ grep -v 'some pattern' -- file.txt

Match integers of a range of numbers of digits (e.g. 2 to 3)

$ myVar="$(printf "S2 E3\nS57 E11\nS131 E51\nS7212 E3\n")"
$ echo "$myVar"
S2 E3
S57 E11
S131 E51
S7212 E3
$ echo "$myVar" | grep -E "S[0-9]{2,3} "
S57 E11
S131 E51

Match http URLs in a text file (see ref)

$ cat input.txt | grep -Eo "(http|https)://[a-zA-Z0-9./?=_%:-]*" | sort -u

gunzip

See #gzip.

gzip

Transform a sql.gz archive into a sql.xz archive.

$ gunzip -c archivo.sql.gz | xz -z - > archivo.sql.xz

iftop

Show data traffic (i.e. bandwidth usage) on network interface eth0. Note: consider using in tandem with nethogs.

$ sudo iftop -i eth0

iotop

Show disk write rates for a given process by PID.

$ iotop -p PID

List accumulative (-a) disk read/write rates for all tar and xz processes via process PIDs (-p), updating every 10 seconds (-d10).

$ sudo iotop -ad10 $(pgrep "tar|xz" | xargs -I "{}" echo -n "-p {} ")

ip

Show available network interfaces.

$ ip link show

ImageMagick

Convert a PNG file into a JPEG at 90% quality.

 $ convert input.png -quality 90 output.jpg

Convert a GIF into a set of frames (Note: May fail with some optimized GIF formats)

$ convert Year_2038_problem.gif output%02d.gif

jdupes

List duplicates in DIR greater than or equal to 100MB.

$ jdupes -X size+=:100MB DIR -r

List duplicates within DIR1 (not following subdirectories) and within DIR2 (following subdirectories)

$ jdupes -X size+=:100MB DIR1 -R DIR2

List duplicates within DIR1, DIR2, and DIR3 recursively, listing duplicates of DIR1 first

$ jdupes -r -O DIR1 DIR2 DIR3

journalctl

Show previous 1 hour of logs:

$ journalctl --utc --all --output=short-iso --since=-1h

Show logs since 2023-01-10T09:15 and before 2023-01-10T13:00:

$ journalctl --utc --all --output=short-iso --since=\"2023-01-10 09:15\" --until=\"2023-01-10 13:00\"

less

Display file as scrollable buffer..

$ less file.txt

Display file and display live updates.

$ less +F file.txt

Display file while truncating display of long lines.

$ less -S file.txt

Display file while interpreting ANSI color codes (e.g. $ jq -C '.' file.json | less --RAW-CONTROL-CHARS)

$ less --RAW-CONTROL-CHARS

ls

Note: assumes ls from GNU Coreutils 8.32

List all files, sorted by ISO-8601-style date.

$ ls -alh --time-style=long-iso | sort -k6,7
-rw-rw-r-- 1 baltakatei baltakatei  88K 2005-08-19 19:18 file1
-rw-rw-r-- 1 baltakatei baltakatei 930K 2010-07-28 02:01 file2
-rw-rw-r-- 1 baltakatei baltakatei 2.4M 2016-05-18 14:52 file3
-rw-rw-r-- 1 baltakatei baltakatei 7.2K 2021-05-11 15:29 file4

lsof

Display all files opened by a process by a single PID.

$ lsof -p PID

Display all files opened by a process by name (e.g. xz)

$ while read -r line; do lsof -p "$line"; printf "\n"; done < <(pgrep xz)

mail

Reference: https://devanswers.co/you-have-mail-how-to-read-mail-in-ubuntu/

Commands:

$ mail   # start mail
& h$     # list latest messages
& 5      # read message 5
& d 1    # delete message 1
& q      # quit mail

Send mail to self:

$ mail -s "I'm in your base" -- "$(whoami)" < <(printf "Killing your dudes.\n")

Delete all mail:

$ mail -N
& d *
& quit

make

See GNU Make

Compile source code according to a file named Makefile

$ make

Note, even with GNU make, column 1 indentations REQUIRE a tab (i.e. `\t`), not a space (`\s`).

Compile multiple source code files with a single `make all` command. `Makefile` contents:

all: program1 program2

program1: program1.c
    gcc -o program1 program1.c

program2: program2.c
    gcc -o program2 program2.c

mdadm

Software RAID manager.

Check status of RAID device /dev/md0.

$ sudo mdadm --detail /dev/md0

Check status of all RAID devices.

$ sudo cat /proc/mdstat

mimetype

Get file mimetype

$ cat file.jpg | mimetype --stdin

mpv

Use newline-delimited stdin list of file paths as playlist.

$ find ~/Music/ -type f | mpv --playlist=-

Enable shuffle

$ mpv --shuffle

Settings for fast playback (e.g. 2x)

$ mpv --af=scaletempo=stride=15:overlap=1:search=15'

Play video with subtitle file

$ mpv --embed-subs="$filepath" video.mp4

neofetch

Show system information without art

$ neofetch --off

nethogs

Show data traffic (i.e. bandwidth usage) by process on network interface eth0.

$ sudo nethogs eth0

Toggle between data rates and total data amounts with `m`.

notify-send

Note (Debian): Installed via the libnotify-bin package.

Create a system notification:

$ notify-send "title" "body"

openbox

A window manager for LxQt and Lubuntu.

Reload configuration files.

$ openbox --reconfigure

pandoc

Convert markdown text file into mediawiki code.

$ pandoc -f markdown -t mediawiki -o output.wc input.txt

par2

Create parity files of archive.tar.xz with default settings.

$ par2 create archive.tar.xz.par2 archive.tar.xz

parallel

See GNU parallel

Duplicate a set of directories (non-recursively) (e.g. home sub-directories). Metadata not copied.

$ find "$HOME" -mindepth 1 -maxdepth 1 -type d | parallel mkdir "$HOME/{}"

Hash every file in the home directory.

$ find "$HOME" -type f | parallel --jobs="" sha256sum '{}'               # use all CPUs
$ find "$HOME" -type f | parallel --jobs="25%" sha256sum '{}'  # use at most 25% of CPU cores
$ find "$HOME" -type f | parallel sha256sum '{}'               # 

Run a thread for every item in an array.

$ myArray=("jan"); myArray=("feb"); myArray=("mar");
$ declare -p myArray
declare -a myArray=([0]="jan" [1]="feb" [2]="mar")
$ parallel echo '{}' ::: "${myArray[@]}"
jan
feb
mar

Supply different arguments for each job with an `--arg-file`.

$ printf "foo\tbee\n" >> args.txt;
$ printf "bar\tboo\n" >> args.txt;
$ printf "baz\ttax\n" >> args.txt;
$ parallel --col-sep '\t' --arg-file args.txt echo '{2}' '{1}';
bee foo
boo bar
tax baz

Avoid OOM by suspending jobs on low memory via --memsuspend 512M (e.g. 512 mebibytes), which suspends job if less than 2 * 512 = 1024 mebibytes memory free. If only one job remains, it will not suspend.

$ seq 1 10 | parallel --memsuspend 512M echo '{}';

ps

Show process PIDs and full commands.

$ ps -eo pid,args

pdftk

Combine PDFs

 $ pdftk doc1.pdf doc2.pdf doc3.pdf cat output output.pdf

pgrep

Exit early if a specific process (e.g. yt-dlp) is already running.

 $ if pgrep "yt-dlp" 1>/dev/random 2>&1; then exit 1; fi;

printf

GNU Coreutils

Round a float to nearest integer:

 $ myVar="14.28571"; printf "%.0f\n" "$myVar";
 14
 $ myVar="14.28571"; printf "%.2f\n" "$myVar"
 14.29
 $ myVar="14.28571"; printf "%.1f\n" "$myVar"
 14.3
 $ myVar="-14.28571"; printf "%.2f\n" "$myVar"
 -14.29
 $ myVar="28.57142"; printf "%.2f\n" "$myVar"
 28.57
 $ myVar="28.57142"; printf "%.0f\n" "$myVar"
 29

Note, GNU Coreutils printf uses “round to even” (i.e. “Bankerʼs rounding”) for cases when 5 must be rounded.

 $ myVar="5.5"; printf "%.0f\n" "$myVar"
 6
 $ myVar="6.5"; printf "%.0f\n" "$myVar" 
 6
 $ myVar="7.5"; printf "%.0f\n" "$myVar"
 8
 $ myVar="8.5"; printf "%.0f\n" "$myVar" 
 8

Print integer with leading zeroes. (see ref)

$ n=7; printf "%05d\n" "$n";
00007

Print a bash array (see ref)

$ declare -a my_array; my_array+=("jan"); my_array+=("feb");
$ printf '%s\n' "${my_array[@]}";
jan
feb

Print a progress bar

#!/bin/bash

total_iterations=100
current_iteration=0

while [ $current_iteration -lt $total_iterations ]; do
    # Your actual loop content goes here
    sleep 0.1 # This is just an example, replace with your actual task

    # Calculate progress percentage
    progress_percentage=$(( 100 * current_iteration / total_iterations ))

    # Print progress percentage without causing scrolling
    printf "\rProgress: %3d%%" $progress_percentage

    # Increment the iteration counter
    current_iteration=$(( current_iteration + 1 ))
done

# Print a newline character to move to the next line after the loop is done
echo

C stdio.h

Print an int as a hexadecimal

int a=17; printf("%x\n",a);

Print an int as a binary (glibc >2.35, check via $ ldd --version)

#include <stdio.h>
int main()
  int a=1023; printf("%b\n",a); return 0;

When compiled with gcc-12 (previous versions throw errors) and glibc >2.35, this prints:

1111111111

rev

Get a counted list of unique file extensions in the current working directory.

$ find ./ -type f | rev | cut -d'/' -f1 | cut -d'.' -f1 | rev | sort | uniq -c | sort -hk1;

An explanation:

$ find ./ -type f | \  # Get a list of files in current working directory.
  rev | \              # Reverse order of characters within each line.
  cut -d'/' -f1 | \    # Get file name.
  cut -d'.' -f1 | \    # Cut all characters except for those before the final `.` in the filename.
  rev | \              # Restore order of characters within each line.
  sort | \             # Sort for uniq.
  uniq -c | \          # Count and remove duplicates.
  sort -hk1;           # Sort by extension count field of each line.

rsync

Note: These commands assume use of rsync version 3.2.7 protocol version 31, which is available on Debian version 11.

Exclude all dotfiles or dotdirectories at any directory level.

$ echo ".*/**" >> exclude.txt
$ rsync -avu --exclude-from=exclude.txt somepath/SOURCE/ anotherpath/DEST/

Copy all files contained within a directory named SOURCE located within somepath into a directory named DEST within anotherpath, preserving file attributes (e.g. user:group, read/write/execute permissions), and overwriting existing files within DEST if they differ in modification date and/or size from those of SOURCE. The forward slashes after SOURCE and DEST are significant; omitting them may cause the creation of a new directory layer instead of synchronizing the file trees of SOURCE and DEST.

$ rsync -avu somepath/SOURCE/ anotherpath/DEST/

Make the contents of DEST exactly match that of SOURCE, overwriting and deleting files as required in DEST via the --delete-before command, performing all deletions before file copying begins. This is useful for updating a backup of SOURCE at DEST.

$ rsync -avu --delete-before somepath/SOURCE/ anotherpath/somedir/DEST/

Copy files from a local SOURCE to a DEST in a remote user's home directory (e.g. /home/username/DEST/) via the ssh command.

$ rsync -avu -e 'ssh' somepath/SOURCE/ username@hostname:DEST/

Copy files only files containing _small in their filenames from a SOURCE to DEST. This preserves the directory tree of SOURCE.

$ rsync -avu --progress --include '*/' --include '*_small*' --exclude '*' somepath/SOURCE/ somepath/DEST/

Move only the files containing _small in their file names, deleting them from SOURCE if successfully copied to DEST.

$ rsync -avu --progress --include '*/' --include '*_small*' --exclude '*' --remove-source-files somepath/SOURCE/ somepath/DEST/

Recreate full path at destination.

$ rsync -avu -R somepath/SOURCE/ anotherpath/DEST/
$ ls anotherpath/DEST/somepath/SOURCE/

sed

GNU sed manual.

Replace first instance of a string.

$ echo "foo foo" | sed 's/oo/ee/'
fee foo

Replace all instances of a string.

$ echo "foo foo" | sed 's/oo/ee/g'
fee fee

Replace all instances of a string in a file (CAUTION: modifies the file):

$ printf "foo\n" > bar.txt
$ cat bar.txt
foo
$ sed -i 's/oo/ee/g' bar.txt
$ cat bar.txt
fee

Append something to the start of each line (ref):

$ printf "bar\nbaz\n" | sed 's/^/foo/'
foobar
foobaz
$ printf "bar\nbaz\n" | sed 's~^~foo~'   # use ~ instead of / as regex delimiter
foobar
foobaz

Delete blank lines. (see ref)

$ printf "foo\n\nbar\n"
foo

bar
$ printf "foo\n\nbar\n" | sed '/^$/d'
foo
bar

Remove an initial `./` from the start of file lists produced by `find` whether newlines or NULL chars are used as list delimiters. Example: sumdir v0.1.2

$ sed -E 's/(^|\x00)\.\//\1/g'

Print specific lines of a file. (i.e. get a specific line from a file)

$ some_command | sed -n '2p'  # prints line 2 of standard input
$ sed -n '2p' file.txt        # prints line 2
$ sed -n '2q;d' big_file.txt  # prints line of a very large file.
$ sed -n '2p;'
$ sed -n '2,5p' file.txt      # prints lines 2 through 5 inclusive.
$ sed -n '2;5;' file.txt      # prints only lines 2 and 5.

Substitute special characters

$ echo "2 * 5 = 8" | sed -E -e 's/*/x/'    # doesn't work because asterisk is special regex
sed: -e expression #1, char 6: Invalid preceding regular expression
$ echo "2 * 5 = 10" | sed -E -e 's/\*/x/'  # works

ssh

Connect to a local machine's Syncthing instance via firefox.

$ firefox 127.0.0.1:8384

Connect to a remote server's Syncthing instance via ssh port forwarding and firefox.

$ ssh -L 127.0.0.1:8388:127.0.0.1:8384 user@hostname
$ firefox 127.0.0.1:8388

Get a host's SSH fingerprint

$ ssh-keygen -l -f /etc/ssh/ssh_host_ecdsa_key.pub

Configure gnupg, ssh, and smartcard on macOS

Install necessary homebrew packages

% brew update && brew upgrade;
% brew install gnupg pinentry-mac

$ gpg -K should have a line like this with [A]:

ssb>  rsa4096/0x5F9D26B9A598A2D3 2018-05-16 [A] [expires: 2026-07-07]

Configure GnuPG to use pinentry-mac

% which pinentry;
/opt/homebrew/bin/pinentry-mac;
% which pinentry-mac >> ~/.gnupg/gpg-agent.conf;

Configure GnuPG to be able to talk to ssh by:

Adding these lines to ~/.gnupg/gpg-agent.conf:

default-cache-ttl 300
max-cache-ttl 999999
enable-ssh-support

and exporting these environment variables to your shell (probably ~/.zshrc as of 2024)

export GPG_TTY="$(tty)";
export SSH_AUTH_SOCK="$(gpgconf --list-dirs agent-ssh-socket)";

Export the ssh public key from your OpenPGP key via GnuPG:

Get public key line to add to remote machine

$ gpg --export-ssh-key YOUR_KEY_ID > my_gpg_ssh_pubkey.txt

This should look like:

$ cat my_gpg_ssh_pubkey.txt 
$ ssh-rsa AAAAB3NzaC1yc2EAAAA… user@host

Add ssh pubkey to remote machine by

running this command remotely (via a preëxisting ssh session or by visiting the remote machine physically)

remote$ echo "ssh-rsa AAAAB3NzaC1yc2EAAAA… user@host" >> ~/.ssh/authorized_keys

or by running this comand locally:

$ gpg --export-ssh-key YOUR_KEY_ID | ssh user@remote 'cat >> ~/.ssh/authorized_keys'

Restart gnupg to apply configuration changes:

$ gpgconf --kill gpg-agent;
$ gpgconf --launch gpg-agent;

Test

$ ssh user@remote

If you never have to use the server's password for the user user, then you succeeded.

sort

Sort part of a checksum file while ignoring some initial lines (e.g. a checksum file generated by sumdir). Sorts every line except for the first three lines which it leaves at the top; the output is written to /tmp/0.txt. Uses the -k2 (i.e. "key 2") option of sort which says to sort by the file name, not the hash (hash is first whitespace-separated entry, file name is the second).

file=.SUMSHA256--20230126T050458+0000; ( cat "$file" | head -n3; cat "$file" | tail -n+4 | sort -k2; ) > /tmp/0.txt

Sort on the third field of comma-delimited lines

$ printf "1,foo,kobo\n2,bar,kaela\n3,baz,zeta\n" | sort -t',' -k3
2,bar,kaela
1,foo,kobo
3,baz,zeta

Remove duplicate lines without sorted result (preserving first copied unique line). (see ref)

$ myVar="$( printf "gundam\ninuyasha\ngundam\nbleach\ngundam\nnaruto\ngundam\n" )"
$ echo "$myVar" | cat -n | sort -uk2 | sort -n | cut -f2-
gundam
inuyasha
bleach
naruto

Preserving last unique copied line.

echo "$myVar" | tac | cat -n | sort -uk2 | sort -n | cut -f2- | tac
inuyasha
bleach
naruto
gundam

stdbuf

A GNU Coreutils program that controls how stdin, stdout, and error data is passed in and out of a program.

Read input from stdin and pass through output to stdout without any buffering.

$ stdbuf -i0 -o0 -e0 command

Example: Continuously filtering journalctl output to capture apache-access lines while discarding the first 7 space-delimited fields of each line. If stdbuf is not used in this type of scenario, tr and cut may fail to immediately display important lines as they arrive from journalctl, choosing to wait until a buffer is filled before displaying them (defeating the purpose of the --follow option of journalctl).

#!/bin/bash
journalctl --all --output=short-iso --since=-7d --follow |\
  grep --line-buffered -Eiv " 404 " |\
  grep --line-buffered "apache-access" |\
  stdbuf -i0 -o0 -e0 tr -s ' ' |\
  stdbuf -i0 -o0 -e0 cut -d' ' -f8- -

strace

Read stderr of a backgrounded and disowned process with process ID pid.

$ strace -p "$pid" -e trace=write

su

Open a shell as root.

alice@host: sudo su -
root@host: whoami
root

Open a bash shell as another user, e.g. www-data.

alice@host: whoami
alice
alice@host: sudo su - www-data -s /bin/bash
[sudo] password for alice:
www-data@host: whoami
www-data

sumdir

A script by Christopher Lovejoy (used with checkdir). Source at GitHub.

Create checksum of files in working directory recursively, excluding files with names: ending in .asc or .ots and files starting with .SUM. Resulting file has pattern: .SUM${digest_name} (e.g. .SUMB2--20230128T013153+0000).

$ sumdir -a sha256 -r -x "*.asc" -x "*.ots" -x ".SUM*"
$ sumdir -a b2 -r -x "*.asc" -x "*.ots" -x ".SUM*"

tar

See File compression notes.

Extract a compressed archive (e.g. .tar.xz) in the current working directory.

$ tar -xf archive.tar.xz

Extract to a different directory some/path/. (The positioning of -C is important.)

$ tar -xf archive.tar.xz -C some/path/

tee

Echo stdout to stderr (ref):

$ echo "This is standard error" | tee /dev/stderr | sed 's/error/out/g'
This is standard error
This is standard out

top

View process, sorted by CPU usage

$ top

Shortcuts:

• Arrow keys & page up/down: Navigate through the displayed list in the Task area.
• q: Finish the top with the q-key.
• S-p: Sort the processes by CPU usage.
• S-m: Sort the processes by memory (%MEM) usage.
• S-t: Sort the processes by running-time.
• S-n>: Sort the processes by process ID.
• t: Changes the display of the CPU usage in the summary section.
• m: Changes the display of memory usage in the summary section.
• S-r: Sort the processes in ascending order instead of descending (default).
• c: By pressing c, the 'Command' column shows the entire path from which the processes were started.
• S-v: Shows the parent / child process hierarchy.
• k: Prompts for a process ID and closes the specified process. By default, SIGTERM is used for a graceful shutdown of the process. For a forced shutdown, you use SIGKILL.

tr

Remove unwanted character sets

Keep only printable characters and spaces from a string.

/bin/bash
name="message:おはよう　ございます.";
name_new="$( printf "%s" "$name" | tr -dc '[:graph:][:space:]' )";
printf "%s\n" "$name";
printf "%s\n" "$name_new";

This results in:

message:おはよう　ございます.
message:.

unar

Install on a Debian system via apt.

$ sudo apt update && sudo apt install unar

Decompress a rar archive.

$ unar archive.rar

unzip

For .rar files, see #unar.

Unzip to directory foo

$ mkdir foo
$ unzip -d foo archive.zip

Unzip archives containing file names encoded in non-English encodings:

Shift JIS Japanese encoding.

$ unzip -O shift-jis archive.zip

Simplified Chinese characters encoding.

$ unzip -O gb18030 archive.zip  # GB 18030 is a superset of GBK. Try this first.
$ unzip -O gbk archive.zip      # GBK an extension of GB 2312.
$ unzip -O gb2312 archive.zip   # GB 2312 deprecated in 2017

Big5 Traditional Chinese characters encoding.

$ unzip -O big5 archive.zip

veracrypt

Mount a volume.

$ veracrypt volume.hc

Unmount all volumes.

$ veracrypt -d

If you get an error message resembling Error: umount: /media/veracrypt1: target is busy, then identify the offending process with lsof:

$ lsof | grep '/media/veracrypt1'

Unmount a specific volume.

$ veracrypt -d volume.hc

Create a volume.

$ veracrypt -t -c

wondershaper

Limit bandwidth of network interface eth0 to 5000kbps download and 1000kbps upload.

$ wondershaper eth0 5000 1000

Clear wondershaper limits.

$ wondershaper clear

yt-dlp

For all options, see yt-dlp GitHub page here.

See example wrapper script here (bkytpldl-generic v4.1.1).

Delay between downloads

$ yt-dlp --sleep-requests 2

Remember downloaded videos to avoid redownload attempts.

$ yt-dlp --download-archive some/path/history.txt "$URL"

Randomize order in which playlist items are downloaded.

$ yt-dlp --playlist-random "$URL"

Handle File name too long error by limiting long fields by byte count.

$ yt-dlp -o '%(title).140B.%(ext)s' '<url>'  # limits title to 140 bytes
$ yt-dlp -o '%(title)s.%(ext)s' '<url>'      # may fail if title too long

Download lowest quality.

$ yt-dlp -S '+size,+br'

Write automatic subtitles of a YouTube video to a .vtt file:

$ yt-dlp --write-subs --write-auto-subs https://www.youtube.com/watch?v=bbkUn0o3L1Y

Parse such a .vtt file in order to extract the text (reading every 8th line with an offset)

clear; offset=1; cycle=8; n=0; {
  while read -r line; do
    if [[ ! $((n % cycle)) -eq "$offset" ]]; then
      ((n++)); continue; fi;
    printf "%s\n" "$line";
    ((n++));
  done < Unicode\ and\ Byte\ Order\ \[bbkUn0o3L1Y\].en.vtt;
  echo "STATUS:Done." 1>&2;
} | grep -v "^$";

xargs

Convert several lines of stdin into words. This may be useful if a command needs to perform an operation on all items in a long newline-delimited list as argument parameters instead of standard input. The following expressions are equivalent calls of ls to list the files foo, bar, and baz:

$ printf "foo\nbar\nbaz\n" | xargs -d '\n' ls -alh;
$ ls -alh foo bar baz;

Note: -d '\n' requires that only newlines are used to separate (i.e. delimit) arguments. The -d option is necessary in newline-delimited lists because xargs will split lines on whitespace characters such as the space character. For example, printf "1 qux\n2 quux\n3 corge\n" | xargs -d '\n' ls -alh; will not apply ls -alh to the three files 1 qux, 2 quux, and 3 corge, but instead will erroneously use six other files 1, qux, 2, quux, 3, and corge.

See also

• Wikipedia:Useful CLI commands
• Useful CLI commands/prv

External links

• bk4, reboil.com