Kelvin Tay

After 8+ years as a software engineer, I made a career switch in 2021. I joined CircleCI as a support engineer.

As I enter my 3rd year with CircleCI, I thought it may be good to reflect on this career change so far.

I noticed there are many literatures for support engineers looking to switch to software engineering / development. However, there aren't many opinions on the other direction.

I hope this can be helpful for folks deliberating on similar career changes.

Why I switched

I will be honest here. One of the major reason I switched was due to the salary compensation.

I was drawing a comfortable salary at my previous job. However, my wife and I were hoping to start a family and considered purchasing a home. I was able to negotiate for a better salary in this switch. (I know this is likely a rare case in many countries. For context, I am based in Tokyo, Japan.)

I also wanted to try something (slightly) different.

Covid-19 was much less of a pandemic by 2021, but it prompted me to relook at life and its (un)certainties. One of the interviewers asked me “where do you see yourself in 3 years?”. I took the opposite view and considered what happened in the last 2 years, from 2019 to 2021. So much has changed across societies as we battled Covid-19. For me, the unexpected Covid-19 did spur me to try something different. Life is too short to be doing the same thing for a lifetime.

Critically, I also enjoyed using CircleCI as a customer. Don't get me wrong; My past adventures were with companies I did believe in. However, being a heavy and happy customer of CircleCI, I found it empowering to be in a position to share my enthusiasm with the customers.

How has the journey been

Fortunately, CircleCI is a product for software teams. This meant that, as a support engineer, I am tackling problems with another software engineer on the other side. I can empathize with the customer.

I have found it really rewarding to be able to draw on my past experience, and share best practices and advice with the customers.

Being part of the JAPAC (Japan + Asia Pacific) team, I also get opportunities to brush up my Japanese. To be honest, my wife and Deepl helped me a lot. I never aspire to reach near-native level for Japanese, but this has been a positive bonus.

One of the major perks is also learning new technologies. I get to learn from both our engineers, and also our customers (e.g., their tech stack). Prior to this job, I've never dug deep with Packer, Nomad, Git LFS, and even Windows PowerShell to name a few. (You can find my public repos for testing here)

Would I recommend it?

In my case, I've enjoyed the transition so far. Thankfully, I still get to utilize much of my software engineering knowledge to troubleshoot with customers.

However, like any choice made, technical support as a career comes with challenges too.

I think being able to empathize with customers is critical. Beyond the technical know-how, you'd need to know how to handle delicate communications at times (e.g., system-wide incident).

If you are a software engineer reading this, I would encourage you to consider the switch if:

(Traits)

1. You enjoy communicating with people as much as digging deep into code / configurations.
2. You enjoy debugging, or chasing down root causes as much as implementing new features.

(Strengths)

1. You can empathize with users, and can advocate on their behalf. I think folks contributing to open-source projects may be good candidates.
2. You are patient and can negotiate tough communications.

(Motivations)

1. You would like to be closer to customers and the product.
2. You enjoy opportunities in sharing best practices with customers.
3. You believe in and enjoy the product.

These are just my opinions based on my experience so far. Your mileage may vary.

However, remember that switching does not mean you cannot go back. You can and should re-evaluate this decision along the way.

buy Kelvin a cup of coffee ☕

Here are some random facts about me, in case this ever becomes useful.

1. I am blessed with an identical twin brother. He's the better one!
2. I write with my right hand, but am left-footed football-wise.
3. I became a father to a baby boy late 2023.

buy Kelvin a cup of coffee ☕

A good friend educated me on how he set up a web application to inform the parents minute-by-minute re: the arrival of his baby. 👶

I thought that was a brilliant idea! This way, new parents can push updates, and avoid the thundering of “has the baby arrived yet?” questions. 💡

So, I set about trying to build one, with the following criteria:

• I don't want to spend time and money maintaining any infrastructure. 💰
• I would like to be able to limit the site's visibility to folks I share it with. ㊙️
• I don't need updates to be “real-time”; Some delay or latency is fine. 📠
• The updates should also be in Chinese and Japanese. 🤖
• Building it should be fun! 🎮

“Why don't you just post it on Twitter/X?”

I also wanted to avoid social media for such announcements, if possible.

How did it go?

I ended up with a user experience where:

1. Folks visit the site hosted on https://write.as with password protection.
2. I can post updates via email + image attachments.

In terms of the “tech stack”, this is pretty low-code. I've set it up via:

• Orchestrating all logic within a Pipedream workflow
• Using an email trigger to capture email body and attached images
• Leveraging Deepl's API to translate EN content to ZH and JA
• Uploading images via the Snap.As API
• Publishing via the WriteAs API
• Putting the business logic within a 106-line Python function

You can use Deepl to translate for free (within limits). I also am using a free account on Pipedream, so I'm really thankful for such services.

I spent 2 weekends experimenting before tying this up. However, I really enjoyed building this simple workflow.

buy Kelvin a cup of coffee ☕

I am not a prolific contributor on open-source work in any sense. It has also been a while since I received any issues or pull-requests on my work.

I recently noticed an issue filed by a user on the (unofficial) CircleCI Terraform provider.

The issue raised was something I was aware of but chose not to fix. I was not sure if the impacted feature would be well-used for the bug to be noticed.

That someone reported it helped signal to me this project and feature is depended upon by users. (Thank you @ nagendrasanthosh)

I took a Saturday morning to release the fix, and it was a fun occasion.

I'd like to keep at this policy.

I am not going to jump into fixing any projects if I think it may take more than 1 weekend to resolve. It may sound selfish, but while the project is Free, my time is not; Family will come first. 👪

What is your policy or strategy on protecting your time on OSS work?

buy Kelvin a cup of coffee ☕

My wife and I are expecting our first child this September.

Having been convinced the magic of a topponcino during a parenting class, we decided to make it instead of buying one.

“Oh, it's almost $100 to buy one? Why don't we make one ourselves?”

We thought this (1) would be more fun, (2) allows us to customize as we like, and (3) gives us a greater sense of ownership!

I am hoping I become good at this. No, not parenting, I meant sewing (I kid).

Fabrication

I was introduced to Yuzawaya, by the wife on a Sunday afternoon. Here in Japan, Yuzawaya is the hobby shop, where you can find reasonably-priced fabric, and then of course, Marimekko fabric as well.

Base	Cover

It felt empowering knowing you get to choose what design and material goes into the product. We also had fun people-watching – guessing what others may be creating for their weekend project.

If you are looking to make other items like bibs, Yuzawaya also offers blueprints (they call them recipes). Unfortunately, there was no blueprints for topponcinos.

Sizing things up

Thankfully, the Interweb is an amazing place. I was able to find a blueprint from this organic cotton shop: https://www.fuwarico.com/topponcino/

Still, this presents a bit of a challenge, since we would have to print it on A1-size paper.

The convenience stores here allow prints up to A3 size. Photo-printing services like Kinko's also offer easy A2 prints.

However, I think I would need to fork out $40+ to print in A1. Most services assume you would be printing colour posters at that scale.

Being an engineer, and mainly because I'm skint, I decided to:

1. crop the blueprint to only the intricate portions (within A3 sizes)
2. draw the rest (mainly regular lines) myself onto the fabric

Why pay that $40, if most of the to-be-printed portions can be drawn by myself? I may be imagining things, but I assume my Asian grandparents would be proud.

(Of course, some may argue I could have simply drawn all parts by myself.)

Here's hoping things go well! We'll share the finished product soon, hopefully.

#parenting #sewing

buy Kelvin a cup of coffee ☕

I have released support for the CircleCI Runner resource-class and token in my unofficial Terraform provider for CircleCI, as per v0.10.3.

Developers can now manage the provisioning (and teardown) of CircleCI self-hosted Runners within Terraform. You can explore an example here.

This was a fun challenge, and I wanted to document my journey on this work.

Investigation

Unlike other resources, self-hosted runners are not manageable under the official V2 API. Developers had to use the CircleCI CLI to manage resource-classes and tokens instead.

To port this into my Terraform provider, I was hoping there was a HTTP API available. This way, I can continue using my approach of abstracting the HTTP API away to a Go SDK.

I assumed (wrongly) the CLI was using GraphQL under the hood for Runner operations, as with many others (e.g., for Orbs).

Digging into the source-code, I then realized Runner resource-classes and tokens can be managed via a HTTP API; It was simply not publicly documented, yet.

Glue

After the legwork mentioned above, I tested the HTTP APIs with an OpenAPI (Swagger 2.0) document. This enabled me to generate a Go SDK for Runner APIs.

Why are the Go SDKs separated? Wouldn't it be easier to keep it all in one?

That was something I mulled over for some time indeed. I have documented my reasons for keeping them separate.

Assembly

With the Go SDK published, I “simply” have to then expose the Runner resource-classes and tokens in the Terraform provider codebase.

The main work was done within a pull request here. This also included acceptance tests, and examples.

Ultimately, I noted self-hosted (machine) runners are also available for CircleCI's Server customers (i.e., self-hosted CircleCI). We would want to extend and ensure this addition can be used by platform teams using CircleCI Server.

It turned out that the Runner API:

• is via https://runner.circleci.com for CircleCI Cloud
• is via https://YOUR-SELF-HOSTED-DOMAIN for CircleCI Server

Thankfully, this was a quick patch. I was also able to verify this fix against my own CircleCI Server instance.

Things I learnt

This feature was satisfying for me to build, and I had many learning points along the journey.

1. Read the code: This feature would not have been completed if I did not dig deeper into the publicly-available source code 📖

2. Keep trying: I am still learning (and failing) at Go. However, I think it is important to keep trying and learning. Keeping this source code open-sourced forces me to keep myself honest too about my lack of knowledge. For fellow engineers out there, let's keep at it! 🤓

#terraform #circleci #runner #go

buy Kelvin a cup of coffee ☕

Is it really though?

I apologize for the misleading title. I've done that intentionally, much like how DHH and Fireship may have titled their responses to the Amazon Prime Video team's recent engineering blog post (Mar 2023).

If you skimmed through these pieces, you may conclude hastily that going Serverless is going to hurt your team and product.

via GIPHY

Having worked with serverless technologies at a past career, I have seen its pros and cons. This is much like any other approach, whether monoliths or micro-services.

Let's take a walk.

Would AWS really tell us that “We are totally wrong about Serverless all along” if they still have much at stake to sell you serverless technologies?

In addition, the Amazon Prime Video team had to re-architecture just one of their services (the monitoring service, specifically).

Overtime, business and technical challenges can evolve. Logically, as software engineers, we should also reevaluate our solutions overtime (e.g., architecture) to verify if they still meet the requirements well. I think it is important not to be married to one idea or paradigm as an engineer.

Also, Amazon Prime Video is a huge product, in terms of their user traffic. The scaling challenge with their monitoring service will match that.

Unless your product is similar in scale, I'd suggest calming down before hurrying to re-architecture anything. Watch something on Amazon Prime, perhaps :)

If anything, I do hope this experience from the Amazon Prime Video's engineering team helped provide some useful feedback to AWS internally.

buy Kelvin a cup of coffee ☕

I have no underlying data to back this up, but the consensus seems to be that a technical support career is not that coveted.

Most aspiring software engineers would prefer to jump into software development than start off in technical support. I was also rejected by a coding bootcamp when I offered to give a guest talk about a career in technical support. (It's a shame.)

I can understand that.

I am considering my next steps in becoming a staff support engineer. I would like to remain on the technical path if possible.

I had tried looking up resources for career advancement in customer-facing engineering roles. However, this has been challenging indeed.

If you have any recommendations of similar resources like https://staffeng.com/book/ for technical support engineers, please let me know!

#individualcontributor #career #engineering

buy Kelvin a cup of coffee ☕

Many CI/CD providers, like GitHub Actions and CircleCI, offer the options to run your CI/CD job using Docker images today.

This is a useful feature since you can ensure your job is always running with the same pre-installed dependencies. Teams may choose to bring their own Docker image, or use readily-available community images on Docker Hub for instance.

One challenge is that the Docker image in use may not have been intended for use in CI/CD automation. Your team may thus find yourselves trying to debug puzzles like:

• I'm pretty sure XYZ is installed. Why does the CI/CD job fail to find XYZ?
• Why is the FOOBAR environment variable different from what we have defined in the Docker image?

Docker image 101

When you execute the docker container run ... command, Docker will, by default, run a Docker container as a process based on the ENTRYPOINT and CMD definitions of your image. Docker will also load the environment variables declared in the ENV definitions.

Your ENTRYPOINT and CMD may be defined to run a long-running process (e.g., web application), or a short process (e.g., running Speccy to validate your OpenAPI spec). This will depend on the intended use of your image.

In addition, Docker images will be designed to come with just-enough tools to run its intended purpose. For example, the wework/speccy image understandably does not come installed with git or curl (see Dockerfile).

Docker images may also be published for specific OS architectures only (e.g., linux/amd64). You will want to confirm which OS and architecture the image can be run on.

These are important contexts, when designing CI/CD jobs using Docker images.

Understanding Docker images for CI/CD

Generally, for CI/CD automation, your job will run a series of shell commands in the build environment.

CI/CD providers like GitLab CI and CircleCI achieve this by override your Docker image's entrypoint with/bin/sh or /bin/bash when executing them as containers.

This is why you would want to use the -debug tag variant for Kaniko's Docker image when using in CircleCI for instance.

Additionally, your Docker image may not come with the required tools for your CI/CD automation. For example, you would require git in order to clone the repository as part of the CI/CD job steps.

Debug Cheatsheet

With this information in mind, here is a list of commands you can run locally to debug your chosen Docker image.

# inspect the "built-in" environment variables of an image
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Env"
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=en_US.UTF-8",
  "PYSPARK_PYTHON=python3",
  "SPARK_HOME=/home/glue_user/spark",
  "SPARK_CONF_DIR=/home/glue_user/spark/conf",
  "PYTHONPATH=/home/glue_user/aws-glue-libs/PyGlue.zip:/home/glue_user/spark/python/lib/py4j-0.10.7-src.zip:/home/glue_user/spark/python/",
  "PYSPARK_PYTHON_DRIVER=python3",
  "HADOOP_CONF_DIR=/home/glue_user/spark/conf"
]

# check the default entrypoint
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Entrypoint"
[
  "bash",
  "-lc"
]

# check the default cmd
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Cmd" 
[
  "pyspark"
]

# check tools installed
$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "git --version"
...
git version 2.37.1

$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "python --version"
...
Python 2.7.18

You can also find an example of this debugging for a CircleCI use-case here: https://github.com/kelvintaywl-cci/docker-executor-explore/blob/main/.circleci/config.yml

#docker #cicd #debug #cheatsheet

buy Kelvin a cup of coffee ☕

If your service exposes a HTTPS API, you may already be documenting it via an OpenAPI spec.

For instance, HSBC provides an OpenAPI spec (read: Swagger) for their Personal Credit Cards API (example).

At a previous career with a company that integrates with financial institutions, we developed the culture to describe our APIs with OpenAPI 3.0.

We had the case where our mobile frontend developers were building the UI features faster than our backend team can develop the API endpoints.

Because we agreed on an API contact using OpenAPI 3.0, we were able to provide mock backend servers for them to test with.

And we did it easily with Stoplight's Prism.

All we needed was to provide our OpenAPI spec, and we had a mock backend ready for our mobile frontend. This unblocked both teams, and we always can rely and refer on our API contract as our agreement terms.

Recently, I have been developing a Terraform Provider. Since the underlying cloud service has a HTTPS REST API, I was able to:

1. Describe the underlying service's API as a OpenAPI spec
2. Generate a Go client SDK based on this OPenAPI spec
3. Mock the service with Prism for some quick acceptance tests

Prism does more than mocking; You can also use it to inspect any discrepancies between your API implementation and the API spec. You can find out more on their GitHub page: https://github.com/stoplightio/prism

Highly recommended for developers who use OpenAPI specs as contracts between components/services!

#openapi #mock #testing #API

buy Kelvin a cup of coffee ☕