Maren

linktr.ee/thwidge

Last week I got major surgery — the kind of surgery where you've been planning this for years and you're asleep for seven hours.

The experience was great and I'm super happy, and I wanted to document one specific part that I found really exceptional. This post won't discuss any details of surgery or anesthesia, though it will reference their existence. Names are made up.

The notable experience started in pre-op. At this stage in pre-op, I'd changed into a gown and had all my belongings taken away, and now a series of nurses and the surgeon and the anesthesiologist were visiting to ask me questions like “are you Maren?”, “what procedure are you having done today?”, “what are you allergic to?”, etc.

After the anesthesiologist, a nurse came and sat down. Paraphrasing, she said, “Hi, my name is Alex, I'm your operating room nurse and the last person you have to talk to before surgery. What's going to happen next is I'll ask you a series of questions, mostly repeats of what you've already been asked, and then you and I will stand up, walk together through that door, through the OR hallway and into your specific operating room. In the operating room, I'll introduce you to everyone on your surgical team, and then you'll lay down on the operating table and the anesthesiologist will put you to sleep. Your surgery will happen there in that room. Does that all sound good and do you have any questions?”

I was surprised, but didn't have any questions. I hadn't even been IV'd yet. Alex asked me her questions, then said “alright let's walk to the OR, follow me.”

We stood, and I followed her through a pair of doors into a busy hallway full of nurses and surgeons and others in fancy surgery outfits. We continued into one specific operating room, and when we got in Alex announced “Alright everyone this is Maren — she's who we're operating on today! Maren you can say hi if you want.” There were about six people in the room. I said hello, and then Alex went around the room and pointed to everyone and said “OK Maren this is (name) and they're going to be (title) on your surgery”, and each person waved and said hi. The last person Alex re-introduced was the physician's assistant of the surgeon — she and I had actually been working closely for months to coordinate surgery and prep. We chatted for a minute, she said some reassuring things, and then Alex said “OK one more thing, this is a formality but it's important. Please describe to the best of your ability, loudly so we can all hear you, what surgery you're getting today.” I did that, and then Alex asked me to lay down on the OR table, I did, and then the anesthesiologist scooted over, IV'd me, and put me to sleep.

Afterwards, I was overwhelmed by how humanizing this particular part of the surgery experience felt for me as a patient. I got put to sleep knowing that everyone who was going to be working on me for the next seven hours had just met me, learned my name, and chatted with me a bit, all while I was completely lucid. In chatting with friends both in and outside of medicine, it sounds like this is not standard procedure everywhere. I wish it were!

I made a status page for myself this morning. The status page indicates whether I'm currently online, as in receiving all my regular phone notifications, or offline, as in taking a break from paying attention to all that. I came up with this plan a month or so ago, and the status page is the last step.

I got a flip phone and a pay-by-the-minute SIM card. I activated the SIM card and then used weird USSD codes to forward calls from my smartphone to the flip phone if the smartphone is unreachable. This means that I can turn off my smartphone (or put it in airplane mode), but still receive calls from it in case there's an emergency.

When I want to go offline, I just put my phone on airplane mode, and open my laptop and run $ go-offline to set the status page. When I go back online, I turn my phone back on and run $ go-online on my laptop. The repository is here but there's barely anything to it.

So why a status page? Why not just turn my phone off sometimes? Why not just work on self control?

I think the main reason is permission. I needed to implement a system in order to give myself permission to disconnect. I feel very guilty when I make myself totally unreachable. Also, If I just turned off my phone, I'm sure that after 12 hours (likely less) someone in my life would become worried and reach out to a friend who lives physically close to me.

Now though, people in my life can check my status page for myself if their messages aren't getting delivered, and the whole situation is less concerning.

I'm offline for the first time right now, and it feels great.

Earlier this year, I got tired of doom-scrolling and made an asocial DIY terminal-twitter for myself. It's called Thoughts and I use it every day. Here's what the default configuration looks like.

Today I realized that it would be super easy to add support for no-configuration deployment on GitHub Pages, so I did that! If you've already been using Thoughts, just run thoughts update to pull down this new feature (and much better default CSS, if you're interested).

To deploy to GitHub Pages, simply enable GitHub Pages for your personal thoughts repo, and then post a new thought (GitHub Pages seem not to go live until a commit has been pushed after enabling Pages).

I'm really excited about this change, because it makes it easier for more people to have their own quiet place on the internet to post their thoughts.

Be well <3

Note: while working on this feature, I also fixed a bug that made thoughts update behave very poorly on MacOS — so sorry about that! It should be better now, but you may have to run thoughts update twice in order to see the fix

Early in lockdown, I made a program for writing short text posts in a terminal and putting them on the internet. It's called Thoughts. It was a super fun experience, and I use it to this day. This is an overly detailed explanation of how it works.

What is Thoughts?

As its README description reads, Thoughts is “a POSIX-compliant shell program for making less-than-a-blog-sized text posts from a terminal.”

Like many, during early lockdown my Twitter usage approximately quadrupled. This led to bird-site burnout, and I wanted a different outlet for spouting off on the internet in a way that was linkable, timestamped, and low-friction. I couldn't find any existing tools that worked the way I wanted, and I desperately needed something engaging to spend all my newly-free time on, so I decided to make the thing.

Before writing code, I knew what I wanted the workflow to look like:

  1. Type a command in a terminal and press enter
  2. A terminal-based text editor opens
  3. Type the text you want to post on the internet
  4. Save and exit the editor
  5. The text is on the internet

I also knew that I wanted Thoughts not to depend on an existing platform to work. There are already lots of ways to tweet from a terminal, and corollaries for other social media platforms too. But the point of Thoughts isn't to enable a user to post something from a terminal — its point is to enable a user to put their thoughts on the internet in a way that's easily shareable but otherwise immune to the freaky virality economy of a platform like Twitter. So Thoughts just emits a web page and gives each post a unique link. This means it's self-hosted, which is unfortunate. It would be great if something like Thoughts existed that was more accessible to non-technical users.

That's more than enough backstory, let's talk about code!

How it works

Thoughts is just POSIX compliant shell code, HTML, CSS, and some AWK. Initially, I didn't imagine that Thoughts would be something anyone else would use, so I didn't put much forethought into its architecture. This was definitely the most enjoyable part of working on it — it was just nothing matters, look it works, wow that's so fun, let's keep going. If I could go back and do it again, I'd do the exact same thing.

Thoughts is comprised of a number of little files. Let's look at them:

A screenshot of the root directory of the Thoughts git repository on GitHub

.gitignore is the gitignore file, LICENSE is the MIT license it uses, and README.md is the readme (and the source of the breaking change referenced in that most recent commit). None of those files are actually part of the program.

.foot.html and .head.html are one of the quirky parts of Thought's design. They're basically everything that will eventually go in the web page that Thoughts outputs, other than the user's actual posts. Head will become the beginning of the HTML document; so it contains meta tags, CSS, and everything else that comes before actual posts; and foot will become the end. Foot mostly contains closing tags and a link to the source code:

<hr>
<p style="text-align:center">
<a href=https://github.com/thwidge/thoughts>source</a>
</p>
</body>
</html>

install.sh

install.sh is the installer, and it looks like this:

#!/bin/sh
set -euf

if [ -z ${1+x} ]; then
  cmd='default'
else
  cmd="$1"
fi

binDir=$HOME/.local/bin
stuffDir=$HOME/.local/share/thoughts

if [ -d "$stuffDir" ]; then
    printf "Thoughts is already installed. Reinstall? [y/n]: "
    read -r reply
    if [ ! "$reply" = "y" ]; then
        echo
	echo "OK, nothing's been installed."
        exit 0
    fi
fi

# copy all the program files to the right places
# and also write the local ignore file
#
# haha I just realized that this doesn't have to
# write the ignore file, it could just rename an existing file
mkdir -p "$stuffDir"/bin
cp parse.awk "$stuffDir"/bin
cp update.sh "$stuffDir"/bin
cp README.md "$stuffDir"
cp .head.html "$stuffDir"
cp .foot.html "$stuffDir"
touch "$stuffDir"/.rawthoughts.html
echo '*' > "$stuffDir"/.gitignore
echo '!thoughts.html' >> "$stuffDir"/.gitignore
echo '!.gitignore' >> "$stuffDir"/.gitignore
echo '!.rawthoughts.html' >> "$stuffDir"/.gitignore
echo '!.head.html' >> "$stuffDir"/.gitignore
echo '!cloudbuild.yaml' >> "$stuffDir"/.gitignore
echo '!Dockerfile' >> "$stuffDir"/.gitignore

mkdir -p "$binDir"
cp thoughts "$binDir"
chmod +x "$binDir"/thoughts

if [ "$cmd" = "another" ]; then
    printf "What's the git clone URL for your existing thoughts repository?: "
    read -r reply
    git clone "$reply" "$stuffDir"/thoughts-temp
    cp -r "$stuffDir"/thoughts-temp/. "$stuffDir"
    rm -rf "$stuffDir"/thoughts-temp
    echo
    echo 'Done! Add $HOME/.local/bin to your PATH'
    exit 0
fi

echo
echo 'Done! Add $HOME/.local/bin to your PATH, and create a git repo: https://github.com/marenbeam/thoughts#first-install'

The logic is:

  1. Check if the user issued ./install.sh or ./install.sh another
  2. Check whether Thoughts has already been installed
  3. Copy all the necessary program files into the correct places in the user's $HOME, and write a gitignore file
  4. If the user issued ./install.sh another, clone their existing thoughts page from a remote git repository

That's it! It's pretty simple, and there aren't many guardrails or choices.

update.sh

update.sh is run when the user wants to update Thoughts to the newest version, and it's broken out into its own program so that it can run the way it needs to. Here's what it does:

#!/bin/sh
set -euf

stuffDir=$HOME/.local/share/thoughts
binDir=$HOME/.local/bin

###
### This script must handle *all* "update" steps
### because it reinstalls its own caller (thoughts itself)
###

cp "$stuffDir"/thoughts-temp/.foot.html "$stuffDir"
echo "copied footer"
cp "$stuffDir"/thoughts-temp/README.md "$stuffDir"
echo "copied readme"
cp "$stuffDir"/thoughts-temp/parse.awk "$stuffDir"/bin
echo "copied parse"
cp "$stuffDir"/thoughts-temp/thoughts "$binDir"
echo "copied thoughts itself"
chmod +x "$binDir"/thoughts
echo "chmod thoughts"

# Handle the possibility of overwriting user's custom CSS
echo
if ! diff "$stuffDir"/thoughts-temp/.head.html "$stuffDir"/.head.html; then
    echo
    echo "WARNING:"
    echo "The CSS in this release is different than what you currently have."
    echo "It could be upstream updates, or maybe you made some customizations."
    echo "Check out the diff above."
    echo
    echo "If you haven't made custom CSS changes, you can safely overwrite and install."
    echo "If you HAVE made CSS changes, just select 'n' and the new CSS will be written somewhere else."
    echo
    printf "DO YOU WANT TO OVERWRITE YOUR CSS? [y/n]:"
    read -r reply
    if [ "$reply" = "y" ]; then
        cp "$stuffDir"/thoughts-temp/.head.html "$stuffDir"
	echo
	echo "CSS overwritten and installed. You're good to go!"
    else
        cp "$stuffDir"/thoughts-temp/.head.html "$stuffDir"/.head-new.html
        echo
        echo "New CSS is in $HOME/.local/share/thoughts/.head-new.html"
	echo "If you want, you can update the CSS yourself with \"thoughts style\""
    fi
fi
rm -rf "$stuffDir"/thoughts-temp
echo
echo "Done updating!"
exit 0

And here's where it's called from the main Thoughts program:

# "update" command
update () {
    git clone https://github.com/thwidge/thoughts.git "$stuffDir"/thoughts-temp
    cd "$stuffDir"/thoughts-temp
    cp "$stuffDir"/thoughts-temp/update.sh "$stuffDir"/bin/update.sh
    chmod +x "$stuffDir"/bin/update.sh
    
    # run update.sh from the new release
    sh "$stuffDir"/bin/update.sh
}

update.sh is broken into its own program for two reasons:

  1. So that it's possible for an update to Thoughts to involve changes to the update script itself (on update, thoughts runs update.sh from the newly cloned repo rather than from the already-installed version)
  2. Because whatever thing is doing the updating must delete the previous version of thoughts itself. I'm not necessarily sure that a shell program can't delete its own file and then continue running, and I'm sure there's a clear answer to this question that involves understanding subshells and processes, but I just made it this way and then moved on. If the way I made it is actually broken, feel free to send me an email.

thoughts

Finally! The actual thing that runs each time you post a thought.

One of the most fun parts of working on Thoughts was using a bunch of coreutils I'd never used before, in ways I'd never used them. I decided that if Thoughts was going to be largely inaccessible to people who weren't already pretty computery, then I'd at least make it as accessible to computery people as possible. That meant I needed to strive for POSIX compliance and edge-case portability everywhere so that Thoughts could work well on any UNIX-adjacent system.

This introduced some really fun constraints. Including:

  • using sh rather than Bash
  • avoiding GNU-specific features in things like sed and awk
  • using only utilities from this list with the sole exception of Git
  • reading lots of angry Stack Overflow comments about the particulars of certain coreutil behavior on weird operating systems from the '90s

Let's look at just one block of code, since a lot of the logic is duplicated in each command:

default () {
    cd "$stuffDir"
    # get an editor so we can type our thought.
    # generate a random temp filename to avoid collisions.
    # who knows what's in there!
    rand=$(date | cksum | tr -d ' ')
    
    "${EDITOR:-vi}" "$rand".txt
    if [ ! -f "$rand".txt ]; then
        echo
        echo "you don't always have to share your thoughts"
        exit 0
    fi
    # If this thought doesn't have a trailing newline, add one
    tail -c 1 "$rand".txt | read -r _ || echo >> "$rand".txt
    
    # replace some newlines with <br>
    # and convert codeblock tag into real one
    # and linkify things outside of codeblocks
    awk -f "$stuffDir"/bin/parse.awk "$rand".txt > temp.txt && mv temp.txt "$rand".txt
    # get the last 4 characters from the file
    # if they are "<br>", delete them.
    br=$(tail -c 5 "$rand".txt)
    if [ "$br" = '<br>' ]; then
        sed '$ s/.\{4\}$//' "$rand".txt > temp.txt && mv temp.txt "$rand".txt
    fi
    thought=$(cat "$rand".txt)
    
    now=$(date +"%I:%M %p | %Y-%m-%d")
    dateHash=$(date | cksum | tr -d ' ')
    blob="<section class=\"thought\"><div class=\"thought-date\"><a class=\"thought-date\" id=\"$dateHash\" href=\"#$dateHash\">\n$now</a></div><div class=\"thought\">\n$thought\n</div></section>\n"
    
    git pull
    echo "$blob" | cat - .rawthoughts.html > "$dateHash".html && mv "$dateHash".html .rawthoughts.html
    cat .head.html .rawthoughts.html .foot.html > thoughts.html
    git add .
    git commit -m "update thoughts"
    git push
    rm "$rand".txt
    echo
    echo "your thoughts have been shared"
}

First, thoughts generates a very-much-not-random value that's misleadingly named rand. We're going to use this value in a few places:

rand=$(date | cksum | tr -d ' ')

This command says “output a checksum of the date but remove everything after the space in the checksum.” We're doing this because we just want a POSIXy way to get a unique-ish integer — we don't care about the actual checksum of the date. Here's what we get without tr -d ' ':

thwidge@uwc:~$ date
Mon 26 Oct 2020 10:07:25 PM EDT
thwidge@uwc:~$ date | cksum
927889335 32

Now that we've got our unique-ish rand integer, we want to open a file <rand>.txt in a text editor. The goal is to start writing in a new text file with a randomish name rather than a hard-coded name, just in case something weird happened and the user has an unpublished thought saved in this directory as a result of a previous crash.

"${EDITOR:-vi}" "$rand".txt
    # If $rand.txt doesn't exist, the user quit without saving or something
    # Handle and exit -- they did not write a thought
    if [ ! -f "$rand".txt ]; then
        echo
        echo "you don't always have to share your thoughts"
        exit 0

The first line "${EDITOR:-vi}" "$rand".txt took me an unreasonably long time to figure out because I was thinking too hard. It says “open rand.txt using the program specified by $EDITOR, or fall back to vi”. For many people this is vim, for others it's nano, for others it's neovim or ed or emacs — great. We can all live in harmony and post our thoughts on the internet 😉

If after we return from the editor $rand.txt does not exist, then we know that the user exited the editor without saving anything. We print a friendly message to the terminal and exit the whole program.

If the user did write a thought, we need to be sure there's a newline at the end of it since not all editors always add newlines to the ends of files (VSCode most notably), but all of our coreutils depend on text files (maybe all files?) ending with newlines. We do that here:

tail -c 1 "$rand".txt | read -r _ || echo >> "$rand".txt

This line says “pipe the last character of the file containing the user's thought into read, and if read doesn't exit with zero then append nothing to the file with echo.”

I honestly don't completely understand how this line is working, and it depends on a few coreutils hacks.

  1. read is waiting for a newline, that's how it works. So I think that this newline test with read works because if the char we've sent it isn't a newline then read won't exit with zero and so the left side of the OR test evaluates to false
  2. If the left side of the OR test evaluates to false, we can just add a newline by using echo to append “nothing” to the file. This will actually append a newline, because that's just how echo works. I'm not sure whether this is officially documented behavior, but you could investigate here if you'd like.

Overall, one of the biggest things I learned while working on Thoughts is that dealing with newlines pretty hard.

Now that we've made sure the user's thought has a newline at the end, we're ready to convert what they've typed into valid html that's ready to get posted on the internet. We use an AWK script to do that here:

awk -f "$stuffDir"/bin/parse.awk "$rand".txt > temp.txt && mv temp.txt "$rand".txt

Originally I was going to do a walk through of this AWK script here, but now I've decided to break it out into another blog post since AWK is its whole own thing!

Much of the time, an unnecessary <br> tag is added to the last line of the post by the AWK script, so we need to check for this and remove it:

br=$(tail -c 5 "$rand".txt)
    if [ "$br" = '<br>' ]; then
        sed '$ s/.\{4\}$//' "$rand".txt > temp.txt && mv temp.txt "$rand".txt
    fi

First we grab the last five characters with tail, because we know that the last character is a newline and and we want the four characters that precede it. If those four characters equal <br>, then we delete the last four characters of the line with sed '$ s/.\{4\}$//'. There's some interesting stuff going on here with how tail thinks about newlines and how sed thinks about newlines, since we're telling sed to do something with the last four characters but telling tail to do something with the last five characters. Like I said above, newlines are pretty hard.

I'm going to cheap out of breaking down the syntax of that sed regex, because it was hard to figure out and I don't have to understand it anymore! 😅 But I won't leave you totally hanging — here's a link to an awesome site with great information about all things relating to Unix coreutils: https://www.grymoire.com/Unix/Sed.html

Finally now we're ready to package our thoughts post up and append it to the existing file that has all this users previous posts in it. We do that here:

thought=$(cat "$rand".txt)
    
now=$(date +"%I:%M %p | %Y-%m-%d")
dateHash=$(date | cksum | tr -d ' ')
blob="<section class=\"thought\"><div class=\"thought-date\"><a class=\"thought-date\" id=\"$dateHash\" href=\"#$dateHash\">\n$now</a></div><div class=\"thought\">\n$thought\n</div></section>\n"

git pull
echo "$blob" | cat - .rawthoughts.html > "$dateHash".html && mv "$dateHash".html .rawthoughts.html

Here, we've already done all the processing we need to to get the actual text that the user typed ready for the web. Now we just need to get it into the HTML file with all the previous thoughts.

First we get the contents of $rand.txt into a variable we can work with more easily (thoughts), then we generate our right-now date stamp (we've already done this previously, but we want to generate a new one in case the user started typing this post a long time ago and now the old date stamp is very inaccurate), then we generate another semi-random hash dateHash which we're going to use as the unique id attribute for linking to this specific thought, and then we package it all into blob with all the surrounding HTML it needs:

<section class="thought"><div class="thought-date"><a class="thought-date" id="274890470929" href="#274890470929">
03:20 PM | 2020-12-05</a></div><div class="thought">
Here's an example of one complete thoughts post. <br>
<br>
This is what the variable <code>blob</code> holds in the above code block.
</div></section>

.rawthoughts.html contains all of the user's previous thoughts posts, so we add this new thought to the file with the old thoughts with this:

echo "$blob" | cat - .rawthoughts.html > "$dateHash".html && mv "$dateHash".html .rawthoughts.html

If you'll notice though, we do a git pull before doing this. This is part of the work of handling the “thoughts can be installed on N computers” feature. If the user has posted thoughts from another computer between the last time they posted from this computer and now, then we need to get that updated .rawthoughts.html from the remote before publishing this new thought. What's more, since we do this right before actually adding the new thought, the user could even have:

  1. Started typing a thought on this computer, but not finished it
  2. Posted a thought from another computer
  3. Come back to this computer and then finished their previously half-finished thought and posted it

And everything should still work! Very nice 🤓

After this, we're almost done publishing this thought. First we package all the thoughts, including the new one, into the final version of the webpage with:

cat .head.html .rawthoughts.html .foot.html > thoughts.html

And then push to remote and clean up a bit:

git add .
git commit -m "update thoughts"
git push
rm "$rand".txt
echo
echo "your thoughts have been shared"

I think this covers most of the interesting parts of Thoughts, and I'm running out of steam on working on this post, so that's all for now! Feel free to send me an email with any thoughts or questions you might have :)

Note: this post is from before I moved my blog to this platform.

I went down a rabbit hole the last few days. As is tradition, the rabbit hole started with ugh, I don't like my website.

This feeling kind of pervades my life. As an ops-skewing gal I both don't know how browsers work and find too much magic hard to swallow. But as a tech-employed gal I feel obligated to have an internet presence with some degree of intentionality. So, a website.

I hadn't ever written HTML until last year. Honestly I'm not sure I'd ever right-clicked inspect until the year before that — though that was certainly the result of my own hardheadedness. I got on the Browsers Are Dumb And The Internet Shouldn't Have Happened train early. I've since decided that browsers are definitely dumb but the internet is important and browsers are what we have so you gotta just do it, too bad.

As I soon came to learn, one person doesn't typically produce a pretty website without deploying a substantial amount of magic. And it is truly magical! Just npm install something and then type your website words and then probably npm something else (I have no idea what I'm talking about) and then npm just poofs fifteen more magical things and then firefox localhost:1234 and oh my goodness where did that stunning vision of a website come from.

But I couldn't do it. I had to know. I had to muck enough in the details that I could trick myself into believing that I knew what was going on when that sweet, sweet 200 came back.

So I went straight for HTML from the start. Since I don't know how browsers work and I'd never written HTML or right-clicked inspect before it was extremely slow going. But I made a thing. And then I remade it. I think I remade it a third time before I realized that what I was calling 'learning' was actually 'iterating uselessly on my brand' and I needed to pull myself together.

So of course I remade it one more time but this time I shamelessly prioritized look and was able to let it sit for a bit.

But something was still eating at me and I couldn't figure out what.

I moved on with my life — hanging out with friends and mothering clusters with my coworkers. Then the pandemic hit and my gaze turned navel and I started fussing with more side projects. One of these side projects was a bash script that got completely out of control and turned into a microblog. It's called Thoughts and there's definitely no “plan” but since you type opinions in vim and get back a website I started actually learning how HTML works.

And then one day, I realized. It was my blog. My blog was making my But What Is It Really alarms go off veeery quietly.

Now don't get me wrong, my blog was fine — a smattering of pandoc and python and bash that made it so I could just type markdown and receive an aesthetically coherent, syntax-highlighted webpage — but when I right-clicked inspect there was so much...stuff. Where was it coming from? What was it doing? How could I sleep when my brand depended on all this magic?

So I went down the rabbit hole. I wanted to be able to type markdown but get dumb HTML back. The kind of HTML that makes you say yep, that's a website.

First I found Txti. I'm still convinced it's one of the best things on the internet, but it doesn't support codeblocks and this is mostly a programming blog so that wouldn't work.

Then I remembered that rwtxt exists and it's honestly beautiful, but: still a decent amount of magic, not clear if you can self-host publicly while limiting public domain creation, and designed for the “wiki” use case. I decided to pass there, too.

Then I remembered that I had just written an AWK function which, when it's low tide and Mercury's in retrograde, turns fake-markdown into very dumb HTML. Maybe I could expand this a bit to cover the other essential markdown things I'd want for a blog, and then I'd have a really opinionated fake-markdown parser that gave me the exact HTML I want. But I'd probably need to open source it and that means I'd have to worry about portability, and I'd have to learn so much more AWK and am I really trying to sink tens of hours into learning AWK right now? Because I'm not sure that's a very Career Oriented Decision but also it's pretty important not to limit my learning based on career utility so maybe I could just—

And then it hit me. Bricks, etc. What if I just...wrote HTML?

So I did. I am right now. It's 2020, and I'm just writing HTML. And honestly, it's perfect.

Thank you to the Recurse Center for featuring my writing in the first issue of Still Computing!

I hope this post might be helpful for someone using Traefik for the first time, someone moving from Traefik v1 to v2, or someone who's getting familiar with Docker compose.

My use case and constraints

I want to host many different things on one box. Currently, the most boring way to do that is with Docker. My previous setup, though technically simple, felt overwhelming because I was holding state in my head rather than in files. Docker would force me to put more system state in files.

Parts

  • Debian installed on the server (aka the host)
  • Docker installed on the host
  • Traefik as an ingress controller for all Docker services
  • Swarm mode enabled on the host
  • Automatic deploy/update with git and cron

Why swarm mode and Traefik? I think swarm gets you most of the declarative things that make deployment easy, without the wild overhead of Kubernetes. I'm using Traefik because it's what I know and it hasn't let me down yet! Also, the SSL story is very straightforward.

Prep

  • Install Debian on the host, and set up SSH
  • Install Docker and Docker compose on the host
  • Enable unattended upgrades on the host
  • Set some good ufw rules on the host
  • Go to your DNS provider, and create an A record for mywebsite.com that points to the IP address of your host

SSH into the host and decide where you want all your Docker service configurations to live. I put mine in ~/docker. They could all go in one docker-compose.yaml` file, but I put mine in different directories because I have unrelated services running on the same host. If you adhere to this framework, then you'll want a parent folder for everything, a folder for the ingress controller configuration, a folder for the website configuration, and a subfolder for the source code of the website itself.

$ mkdir -p ~/docker/traefik
$ mkdir -p ~/docker/mywebsite.com/site

And for our last piece of host-setup, we need to enable swarm mode and create a Docker network for our services to use.

$ docker swarm init
$ docker network create --driver overlay proxy

We're creating an overlay network because this is a swarm node and we'll be deploying swarm services to it. All services will connect to this network so that they can talk to Traefik, and Traefik will be the only thing that can talk to the internet. You can find more information about overlay networks here.

One way to deploy swarm services is to write a docker-compose.yaml configuration for each service, and then deploy them with docker stack deploy. This is well-supported in the docker documentation, so it's what we're going to do.

$ vim ~/docker/traefik/docker-compose.yaml

Now we can really start doing stuff!

Configure Traefik

First we're going to set up Traefik. Paste this configuration into the file you just opened, and edit as necessary for your use case. At the very least, you'll need to change the email address. I've included comments explaining most lines.

version: "3"

services:
  traefik:
    # specify the docker image we're deploying as a service
    image: "traefik:latest"
    # this specifies the name of the network the service will connect to
    networks:
      - "proxy"
    # these commands override service configuration defaults
    command:
      # set the service port for incoming http connections
      - "--entrypoints.web.address=:80"
      # set the service port for incoming https connections
      - "--entrypoints.websecure.address=:443"
      # enable the traefik api. this would be used by the traefik dashboard if we set that up
      - "--api=true"
      # tell traefik that it's connecting to a swarm, rather than regular docker
      - "--providers.docker.swarmMode=true"
      # traefik automatically finds services deployed in the swarm ("service discovery").
      # this setting restricts the scope of service discovery to services that set traefik.enable=true
      - "--providers.docker.exposedbydefault=false"

      ### these three lines configure the thing inside of traefik that's going to get/renew/manage SSL certificates for us.
      ### It's called a "certificate resolver"
      # 'leresolver' ("Let's Encrypt resolver") is just the name we're giving to the certificate resolver.
      # The name you choose can be different.
      # set the email address to give Let's Encrypt. we should give them a real email address whose inbox gets checked by a human
      - "--certificatesresolvers.leresolver.acme.email=myemail@mailbox.org"
      # set the location inside the container to store all certificates
      - "--certificatesresolvers.leresolver.acme.storage=/acme.json"
      # tell the certificate resolver the method we want to use to get an SSL certificate.
      # you can read about challenge types here:  https://letsencrypt.org/docs/challenge-types/
      - "--certificatesresolvers.leresolver.acme.tlschallenge=true"

    # because traefik is the ingress controller and thus must talk directly to the internet,
    # we want to bind ports on the traefik container to ports on the debian host. this does that
    ports:
      # container-port:host-port
      - "80:80"
      - "443:443"
    # make things on the host accessible to the container by mounting them in the container
    # /host/path:/container/path
    volumes:
      # mount the docker unix socket inside the traefik container.
      # this is essential for traefik to know about the services it's sending traffic to.
      # we mount it read-only for security. if traefik were compromised, and the docker socket were mounted read/write,
      # the attacker could send instructions to the docker daemon.
      # you can learn about unix sockets here:  https://en.wikipedia.org/wiki/Unix_domain_socket
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      # mount this file inside the traefik container. this is where SSL certificates are stored.
      # if we don't do this, when traefik reboots (which is guaranteed), we'll lose all our SSL certificates
      - "./acme.json:/acme.json"
    # the deploy block is here because this is a swarm service.
    # other than setting labels, we're using all the swarm mode defaults for this service
    # more information is here: https://docs.docker.com/compose/compose-file/#deploy
    deploy:
      labels:
        # redirect all incoming http requests to https.
        # this will apply to all services sitting behind traefik. for us, that's all services
        - "traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)"
        - "traefik.http.routers.http-catchall.entrypoints=web"
        - "traefik.http.routers.http-catchall.middlewares=redirect-to-https"

        # define a traefik 'middleware' to perform the actual redirect action.
        # more information about traefik middlewares:  https://docs.traefik.io/middlewares/overview/
        # more information about the RedirectScheme middleware:  https://docs.traefik.io/middlewares/redirectscheme/
        - "traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https"

# this is necessary because we're connecting to a pre-existing network that we made ourselves. in this case, the 'proxy' network
networks:
  # the name of the network
  proxy:
    # this tells docker, "Don't make this network yourself, because I've already made it." It's 'external' to docker-compose
    external: true

And that's the main Traefik configuration! This may seem like a lot, but we'll never have to touch this configuration again — even if we deploy 50 unrelated services behind this Traefik instance.

Before starting Traefik, let's create and set permissions for the acme.json file where our certificates will be stored. This file will be full of mission-critical secrets, so it's important to do this right.

$ touch ~/docker/traefik/acme.json
$ chmod 600 ~/docker/traefik/acme.json

And that's actually the end of the Traefik configuration.

One thing that Traefik has is a fancy dashboard. For clarity of configuration, we've not set that up. Since we've not set that up, and we haven't deployed our website yet, we don't have a good way to test our setup at the moment. The best you can do is deploy Traefik and then check to see if it's running. We're deploying with docker stack deploy because this is a swarm service.

$ docker stack deploy --compose-file ~/docker/traefik/docker-compose.yaml traefik
$ docker container ls

If you see one Traefik container running, that's great! You could unplug your server from the wall right now (not recommended), plug it back in, and the Traefik service would automatically come back up as soon as Docker was able to make it happen.

Now let's configure our actual website.

Configure the website

First let's work on the Docker service configuration. Open a new compose file:

$ vim ~/docker/mywebsite.com/docker-compose.yaml

And paste in the following configuration. Again, edit as necessary for your use case:

version: '3'

services:
  nginx:
    # we specify that we want to use the alpine-based nginx image
    image: "nginx:alpine"
    # connect to this network in order to connect to traefik
    networks:
      - "proxy"
    # mount the directory containing the source code for our website inside the container.
    # this is the directory that the default nginx configuration automatically serves content from.
    # by putting our site here, we avoid having to write any nginx configuration ourselves
    volumes:
      - "./site:/usr/share/nginx/html:ro"
    deploy:
      labels:
        # tell traefik that it can automatically "discover" this service
        - "traefik.enable=true"
        # tell traefik that all requests for 'mywebsite.com' should be sent to this service
        - "traefik.http.routers.mywebsite.rule=Host(`mywebsite.com`)"
        # only allow incoming https connections
        - "traefik.http.routers.mywebsite.entrypoints=websecure"
        # tell traefik which certificate resolver to use to issue an SSL certificate for this service
        # the one we've created is called 'leresolver', so this must also use 'leresolver'
        - "traefik.http.routers.mywebsite.tls.certresolver=leresolver"
        # tell traefik which port *on this service* to connect to.
        # this is necessary only because it's a swarm service.
        # more info is here: https://docs.traefik.io/providers/docker/#port-detection_1
        - "traefik.http.services.mywebsite.loadbalancer.server.port=80"

# again, we have to specify that we've already created this network
networks:
  proxy:
    external: true

Now we can do a quick test to see whether everything's working up to this point.

$ echo 'hello world' > ~/docker/mywebsite.com/site/index.html
$ docker stack deploy --compose-file ~/docker/mywebsite/docker-compose.yaml mywebsite

Wait for 30 seconds, just for good measure. Consider making some tea! Then, visit mywebsite.com in a browser, or on your local machine:

$ curl https://mywebsite.com

If you get a response (or a page) containing only hello world, success!

Now we can do the last step: setting up automatic deployments with GitHub and cron. If you don't already have a static site you'd like to use for this, you can use this template to start with.

Automatic deployment

Our end-goal workflow for making changes to our site is:

  1. Make changes to our website on our local machine
  2. Assuming our source code is in a public repo on GitHub, commit our changes and run git push
  3. At the top of the next hour, our changes are visible on the internet

We're going to use a cron job running on the host to achieve this. This is a pretty funny combination of new computer (traefik, swarm mode, etc.), and old computer (cron). But do you want to set up a whole CI pipeline for a personal, static website? Me neither! I think cron is perfect for something like this.

We're going to skip over creating a new repo and just work with this template which you should absolutely feel free to use for your own website!

Assuming you've forked my repo, or are otherwise set up with a git repo you'd like to use, now we just need to set up a cron job on our host that'll pull the repo each hour and copy it into ~/docker/mywebsite.com/site for Nginx to serve.

First, SSH into the host. Then:

$ mkdir ~/cronjobs
$ mkdir ~/.mywebsite.com
$ vim ~/cronjobs/update-my-website-dot-com.sh

In the file you just opened, paste the following:

#!/bin/bash
cd ~/.mywebsite.com/mywebsite.com
git pull
# we only want to give nginx the files that we actually want to serve.
# we include the --delete flag so that if we permanently remove a file from our site's source code,
# it's removed from the directory that nginx is serving.
# basically, a true "sync" with rsync requires the --delete flag
rsync -a --delete --exclude '.*' --exclude 'README.md' --exclude 'LICENSE' . ~/docker/mywebsite.com/site/

Make the update script executable, and for good measure be sure rsync and git are installed:

$ chmod +x ~/cronjobs/update-my-website-dot-com.sh
$ sudo apt update
$ sudo apt install rsync git

Now get the repo onto the host, and into the right place — we only have to do this once.

$ cd ~/.mywebsite.com
$ git clone https://github.com/thwidge/mynamedotcom.git

Run the update script once manually to sync the repo right now:

$ ~/cronjobs/update-my-website-dot-com.sh

Finally, cron it:

$ crontab -e

And in that file add the line:

@hourly ~/cronjobs/update-my-website-dot-com.sh

And that's it! You've now got a single node Docker swarm cluster; Traefik accepting incoming requests, routing them to the appropriate service, and programmatically handling SSL provisioning and termination; an Nginx container serving your static site over HTTPS; and a simple cron job reliably syncing and deploying all changes merged to the main branch at the top of each hour.