Skip to main content

Using find -exec

Find has a useful option -exec which can be used to run a command on each file found. It has a couple of slightly obscure details to the syntax:

  • {} is a placeholder for the filename.
  • The command we are running with -exec needs to be terminated with a semicolon as there may be further arguments to find after it. This needs to be escaped as \; or ';', otherwise it will be interpreted as the end of the find command.

It is also possible to pass the -exec argument multiple times

Stringing this together allows us to assemble some elegant one-liners, for example:

# delete log files more than 60 days old
find ./logs -mtime +60 -name '*.log' -exec rm {} \;

# print the name and (array/object) length of all JSON files
find . -name '*.json' -exec printf {}\:\  \; -exec jq '. | length' {} \;

# recursively change permissions on files (ignore directories)
find . -type f -exec chmod 644 {} \;

Pi-Hole

I've been running Pi-Hole for years and I'm a big fan of it. It provides network-level blocking of ads and trackers. Set up Pi-Hole, configure your network to use it as a DNS server and it won't resolve DNS requests for known ad/tracking domains. This is harder to detect than in-browser content blockers and also blocks ads on mobile platforms where ad blockers may not be available because the blocking is applied at the network level, rather than the device.

Pi-Hole can run on a Raspberry Pi (hence the name) - and that's how I run mine - but can also be installed on other hardware (including via Docker). It can also be configured as a DNS-Over-HTTPS proxy using Argo Tunnel so you can use DOH and have DNS-level ad-blocking.

Post JSON to an API with curl

Increasingly this is becoming just a place to store snippets and one-liners that are a bit too long to remember.

curl "https://my-api/endpoint" \
  -X POST \
  -H "Accept: application/json" \
  -H "Authorization: f00ba2" \
  --data '{"my": "payload", "foo": "bar"}' \
  --include

Crontab.guru

It doesn't matter how long I use it for, crontab syntax is one of those things is one of those things I just can't commit to memory. It's not complicated, just somewhat unintuitive. Fortunately crontab.guru is a crontab syntax checker which translates crontab expressions into natural language.

Three useful pathlib snippets

from pathlib import Path
p = Path('/foo/bar')


# Read in a text file
text = (p / 'file.txt').read_text()

# Recursively list all .csv files in directory
csvs = list(p.rglob('*.csv'))

# Iterate files or subdirectories
files = [i for i in p.iterdir() if i.is_file()]
subdirs = [i for i in p.iterdir() if i.is_dir()]

Maybe that's three and a half 🙂

Pathlib Cheat Sheet

>>> from pathlib import Path
>>> f = Path.home() / 'foo.tar.gz'  # same as f = Path('/home/chris/foo.tar.gz')

>>> f.is_file()
True

>>> f.is_dir()
False

>>> f.exists()
True

>>> f.absolute()
PosixPath('/home/chris/foo.tar.gz')

>>> f.as_uri()
'file:///home/chris/foo.tar.gz'

>>> f.as_posix()
'/home/chris/foo.tar.gz'

>>> f.parts
('/', 'home', 'chris', 'foo.tar.gz')

>>> f.suffix
'.gz'

>>> f.suffixes
['.tar', '.gz']

>>> f.parent
PosixPath('/home/chris')

ShellCheck

I write bash scripts from time to time, but I'm certainly not a master of it and bash is a technology which has a variety of gotchas, surprising behaviours and foot-guns. This means the scripting process can be a bit of a minefield. Enter ShellCheck - a linter/static analysis tool for Bash. This is one of those tools I wish I had known about years ago as its great for flagging common mistakes and errors.

Using this locally in the terminal is great, but one of my favourite features is that if you paste your script into the editor on shellcheck.net any error codes raised links to a wiki article with a detailed explanation of why this error is being raised and what problems might be caused by this code. Example.

shonky bash scriptShellCheck telling me how to fix a shonky bash script

Working In Public

Nadia Eghbal (who previously authored the seminal report Roads & Bridges) recently released her first full-length book Working in Public: The Making and Maintenance of Open Source Software. Working in Public presents an updated, more nuanced and more refined take on the challenges faced by open source developers and maintainers. As ever, Nadia's superpower is to take the experiences we all feel in the open source community and package them in a way that makes the topic accessible for a new and much wider audience.

pipx

Poetry is my go-to solution for managing libraries and project dependencies in python, but there are also a variety of python-based CLI tools like aws-cli and tldr that I want globally available. But what if they have incompatible dependencies? No worries - pipx has got your back. Pipx installs packages into an isolated virtual environment for each application, keeping their dependencies separate and your global environment clean. This makes the applications globally available while avoiding the possibility of conflicts in the global environment.

gron

gron makes JSON greppable. I'm late to the party on gron, but its already found itself a home in my toolchain. It flattens JSON into valid javascript statements (useful in itself) and makes it easier to process JSON with standard unix command line tools.