Kelvin Tay

Is it really though?

I apologize for the misleading title. I've done that intentionally, much like how DHH and Fireship may have titled their responses to the Amazon Prime Video team's recent engineering blog post (Mar 2023).

If you skimmed through these pieces, you may conclude hastily that going Serverless is going to hurt your team and product.

via GIPHY

Having worked with serverless technologies at a past career, I have seen its pros and cons. This is much like any other approach, whether monoliths or micro-services.

Let's take a walk.

Would AWS really tell us that “We are totally wrong about Serverless all along” if they still have much at stake to sell you serverless technologies?

In addition, the Amazon Prime Video team had to re-architecture just one of their services (the monitoring service, specifically).

Overtime, business and technical challenges can evolve. Logically, as software engineers, we should also reevaluate our solutions overtime (e.g., architecture) to verify if they still meet the requirements well. I think it is important not to be married to one idea or paradigm as an engineer.

Also, Amazon Prime Video is a huge product, in terms of their user traffic. The scaling challenge with their monitoring service will match that.

Unless your product is similar in scale, I'd suggest calming down before hurrying to re-architecture anything. Watch something on Amazon Prime, perhaps :)

If anything, I do hope this experience from the Amazon Prime Video's engineering team helped provide some useful feedback to AWS internally.

buy Kelvin a cup of coffee ☕

I have no underlying data to back this up, but the consensus seems to be that a technical support career is not that coveted.

Most aspiring software engineers would prefer to jump into software development than start off in technical support. I was also rejected by a coding bootcamp when I offered to give a guest talk about a career in technical support. (It's a shame.)

I can understand that.

I am considering my next steps in becoming a staff support engineer. I would like to remain on the technical path if possible.

I had tried looking up resources for career advancement in customer-facing engineering roles. However, this has been challenging indeed.

If you have any recommendations of similar resources like https://staffeng.com/book/ for technical support engineers, please let me know!

#individualcontributor #career #engineering

buy Kelvin a cup of coffee ☕

Many CI/CD providers, like GitHub Actions and CircleCI, offer the options to run your CI/CD job using Docker images today.

This is a useful feature since you can ensure your job is always running with the same pre-installed dependencies. Teams may choose to bring their own Docker image, or use readily-available community images on Docker Hub for instance.

One challenge is that the Docker image in use may not have been intended for use in CI/CD automation. Your team may thus find yourselves trying to debug puzzles like:

• I'm pretty sure XYZ is installed. Why does the CI/CD job fail to find XYZ?
• Why is the FOOBAR environment variable different from what we have defined in the Docker image?

Docker image 101

When you execute the docker container run ... command, Docker will, by default, run a Docker container as a process based on the ENTRYPOINT and CMD definitions of your image. Docker will also load the environment variables declared in the ENV definitions.

Your ENTRYPOINT and CMD may be defined to run a long-running process (e.g., web application), or a short process (e.g., running Speccy to validate your OpenAPI spec). This will depend on the intended use of your image.

In addition, Docker images will be designed to come with just-enough tools to run its intended purpose. For example, the wework/speccy image understandably does not come installed with git or curl (see Dockerfile).

Docker images may also be published for specific OS architectures only (e.g., linux/amd64). You will want to confirm which OS and architecture the image can be run on.

These are important contexts, when designing CI/CD jobs using Docker images.

Understanding Docker images for CI/CD

Generally, for CI/CD automation, your job will run a series of shell commands in the build environment.

CI/CD providers like GitLab CI and CircleCI achieve this by override your Docker image's entrypoint with/bin/sh or /bin/bash when executing them as containers.

This is why you would want to use the -debug tag variant for Kaniko's Docker image when using in CircleCI for instance.

Additionally, your Docker image may not come with the required tools for your CI/CD automation. For example, you would require git in order to clone the repository as part of the CI/CD job steps.

Debug Cheatsheet

With this information in mind, here is a list of commands you can run locally to debug your chosen Docker image.

# inspect the "built-in" environment variables of an image
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Env"
[
  "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
  "LANG=en_US.UTF-8",
  "PYSPARK_PYTHON=python3",
  "SPARK_HOME=/home/glue_user/spark",
  "SPARK_CONF_DIR=/home/glue_user/spark/conf",
  "PYTHONPATH=/home/glue_user/aws-glue-libs/PyGlue.zip:/home/glue_user/spark/python/lib/py4j-0.10.7-src.zip:/home/glue_user/spark/python/",
  "PYSPARK_PYTHON_DRIVER=python3",
  "HADOOP_CONF_DIR=/home/glue_user/spark/conf"
]

# check the default entrypoint
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Entrypoint"
[
  "bash",
  "-lc"
]

# check the default cmd
$ docker image inspect docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 | jq ".[0].Config.Cmd" 
[
  "pyspark"
]

# check tools installed
$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "git --version"
...
git version 2.37.1

$ docker container run --rm docker.io/amazon/aws-glue-libs:glue_libs_2.0.0_image_01 "python --version"
...
Python 2.7.18

You can also find an example of this debugging for a CircleCI use-case here: https://github.com/kelvintaywl-cci/docker-executor-explore/blob/main/.circleci/config.yml

#docker #cicd #debug #cheatsheet

buy Kelvin a cup of coffee ☕

If your service exposes a HTTPS API, you may already be documenting it via an OpenAPI spec.

For instance, HSBC provides an OpenAPI spec (read: Swagger) for their Personal Credit Cards API (example).

At a previous career with a company that integrates with financial institutions, we developed the culture to describe our APIs with OpenAPI 3.0.

We had the case where our mobile frontend developers were building the UI features faster than our backend team can develop the API endpoints.

Because we agreed on an API contact using OpenAPI 3.0, we were able to provide mock backend servers for them to test with.

And we did it easily with Stoplight's Prism.

All we needed was to provide our OpenAPI spec, and we had a mock backend ready for our mobile frontend. This unblocked both teams, and we always can rely and refer on our API contract as our agreement terms.

Recently, I have been developing a Terraform Provider. Since the underlying cloud service has a HTTPS REST API, I was able to:

1. Describe the underlying service's API as a OpenAPI spec
2. Generate a Go client SDK based on this OPenAPI spec
3. Mock the service with Prism for some quick acceptance tests

Prism does more than mocking; You can also use it to inspect any discrepancies between your API implementation and the API spec. You can find out more on their GitHub page: https://github.com/stoplightio/prism

Highly recommended for developers who use OpenAPI specs as contracts between components/services!

#openapi #mock #testing #API

buy Kelvin a cup of coffee ☕

Background

I passed the HashiCorp Certified: Terraform Associate 002 exam on Jan 14th 2023. This was my first try, fortunately.

In terms of my Terraform experience prior to the exams, I have yet to build any Terraform project at a production level. My recent exposure was mainly around trying to play around with CircleCI’s Nomad client Terraform setup.

Motivations

In my past career, I have used loads of AWS CloudFormation. Yes, I have had some battle scars re: IaC. I always wanted to learn more about Terraform, given that it supports more than 1 cloud platform.

In particular, I am interested in contributing a Terraform provider for CircleCI. This means creating a provider/plugin for creating CircleCI webhooks / scheduled pipelines via Terraform for example.

This above may likely be a lofty ambition, but I thought I’d hedge this by also trying for the certification. Kill two birds in one quarter, as they say.

Breakdown of Score

Content Area	Score (%)
Understand infrastructure as code (IaC) concepts	100
Understand Terraform’s purpose (vs other IaC)	100
Understand Terraform basics	85
Use the Terraform CLI (outside of core workflow)	100
Interact with Terraform modules	50
Navigate Terraform Workflow	100
Implement and maintain state	75
Read, generate and modify configuration	72
Understand Terraform Cloud and Enterprise capabilities	100

Overall, I scored 84% (70% required).

Word of Advice

If you are not using Terraform on a daily basis, practice Terraform CLI commands definitely!

The exam has a few questions where they may test you to type the code / command required. See sample questions here.

I did have a “doh!” moment, since it wasn’t clear how “complete“ the answer should be.

In particular, one question I got was: How would you reference the vpc_id from this module in the code snippet?

To reference the module output, it would be module.aws_vpc.vpc_id for example. I think the full answer should be: vpc_id = module.aws_vpc.vpc_id.

I had only typed module.aws_vpc.vpc_id due to lack of clarity in the question. I think I likely lost points there!

In terms of studying for it, I highly recommend the following Udemy course + practice exam!

• HashiCorp Certified: Terraform Associate Practice Exams 2023
• More than Certified in Terraform

Lastly, there will be a newer exam version (003). I understand this update would remove older questions referring to already-deprecated commands (e.g., taint). So if you can wait, do go for the 003 version instead!

Final Thoughts

It was a fun exam to take, for sure!

Given that it is really about a tool and its ecosystem, I’d highly encourage folks to try out Terraform while practicing for the exam.

Your mileage on the Terraform tool will go a long way if you practice. This way, the exam questions will make way more sense too, as you practice.

buy Kelvin a cup of coffee ☕

Infrastructure as Code (IaC) is not a new concept for many. Prior to joining CircleCI, I have been wrangling with AWS CloudFormation long enough (2 years) to bear some of its pain-points.

Since joining CircleCI, I have been exposed to Terraform through internal and external projects. So far, I really like that Terraform is not tied to specific cloud providers.

I also like that any service can contribute a Terraform provider, thereby allowing users to define your service's resources as code. (Any cloud resource that exposes an CRUD / RESTful API is a good candidates for Terraform.)

Recently, I have been wishing many things to be “Terraform-able”.

As a support engineer, I use Zendesk daily. We create Zendesk macros for canned responses in particular. Overtime, these macros grow, and it may be hard to manage or validate changes to our macros. I do wish Zendesk has a Terraform provider, so we can institute our macros in code.

Recently, I also wished that I could Terraform my resume. Then, I snapped out of it.

When you have a powerful hammer like Terraform, everything looks like nails.

#terraform #infrastructureascode #declarative

buy Kelvin a cup of coffee ☕

Primer

This post reflects my opinions on how I decide to continue staying at the current job, or consider switching.

As such, this is purely from my perspective. You would likely have your own principles and outlook on life and career; If anything, I hope this post motivates you to reflect, rethink, and refresh.

For context, I work a full-time job with no additional side gigs or secondary jobs.

My Guiding Principles

I use the following 3 factors to decide if I should continue with the current job:

1. Compensation
2. Job Fulfillment
3. Work-Life Balance

You likely have your own principles, so you can replace this list with your own!

I find that realistically, most jobs should hit 2 out of the 3 factors at least.

When I find myself seriously contemplating a career change, I like to check this list. If I evaluate that the current opportunity hits only 1 factor (i.e., score of 1/3), it is a good indication that I can do better.

1. Compensation

Without mincing my words, this is just about the main salary. I am not swayed by or overly concerned with stock options, or welfare benefits.

If you are working in Japan, I do encourage you to check out https://opensalary.jp/. This can help you get some idea about what others with similar tech backgrounds are compensated.

2. Job Fulfillment

This can a tough one to judge or measure.

You can go all logical about it (i.e., making a list of the positives and negatives).

Or, you can also simply base it on gut feeling (e.g., say 8 out of 10).

For me, I tend to find my answer by asking myself:

“If someone asks me what I do for a living, how enthusiastic would I be about introducing my current job?”

In terms of motivation, I like to measure how I feel when I get up on a Monday morning.

I like to point out that a good team can also offer so much support at work. Your motivation and fulfillment can be influenced greatly by your colleagues.

3. Work-Life Balance

I like to think that we spend half of our waking hours working. That is, 8 out of 16 hours trading our skills for compensation.

You may work longer than 8 hours on some days due to circumstances. However, if overwork becomes consistent, something is not healthy with your current environment.

Realistically, not everyone is built to grind out 10+ hours everyday. This is not sustainable for yourself, and your team.

You also likely have aspirations outside of work, hobbies you want to pursue, and family members you want to spend time with.

Ensure that you have time for them.

Final Thoughts

This is just a general guide, based on my perspectives.

That said, I understand there can be inertia to “go”; It feels easier to stay, naturally. I have left out any discussions around evaluating the other opportunities available. This is purely focused on evaluating your current job.

My (check)list describes my definition of what work means to me. I would encourage you to think what work means for you and your family.

Happy to connect, and talk about this over coffee if you are in Tokyo!

#career #motivation #principles

buy Kelvin a cup of coffee ☕

Today's communication tools allows us to include links in our writings easily in many ways.

This sentence includes a link within itself, but does not stop you from reading the entire message.

I find such links useful when the linked article is supplementary to your message.

In situations where I want to direct the reader to click on the link explicitly, I find it best to simply show them raw below: https://www.w3.org/DesignIssues/LinkMyths.html

Personally, I find that showing the link raw (as-is) at the end helps:

1. draw attention, especially if the main point is about the linked article
2. to allow readers to inspect the link URL, which can help in credibility too

buy Kelvin a cup of coffee ☕

Preface

In a CircleCI pipeline, workflows run independent of one another. As such, there is no built-in feature to ensure workflow B runs after workflow A.

However, you can still achieve ordering, through some trickery.

💡 You can control the sequence of jobs within a workflow. I recommend you consider if you can combine/merge workflow B into workflow A itself, first. This article is for when you require separate workflows somehow.

How

To achieve ordering, we simply set an approval job as the first job for workflow B.

# contrived snippet of a .circleci/config.yaml

workflows:
  aaa:
    jobs:
      - one
      - two
  bbb:
    jobs:
      - start:
          type: approval
      - next:
          requires:
            - start

Subsequent jobs in workflow B will only run when the approval job is approved. As such, you can “force” a wait, and only approve this job when workflow A is completed.

Note that this requires manual intervention, of course.

However, a benefit in this approach is that your team can take the time to confirm the outcomes of workflow A. For example, workflow A has deployed some infrastructure changes (e.g., terraform apply), and you prefer inspecting these changes before running workflow B.

One Step Further

You can automate this approval, at the end of workflow A, via the Approve a job API.

Specifically, you would need to create a job that does the following:

1. Find workflow B's ID from the current pipeline.
2. Find the approval job's ID from the invoked workflow B.
3. Approve the job.

jobs:
  ...
  approve-workflow:
    parameters:
      workflow-name:
        type: string
        description: workflow name
      job-name:
        type: string
        description: name of approval job in workflow
    docker:
      - image: cimg/base:current
    steps:
      - run:
          name: Find Workflow ID for << parameters.workflow-name >>
          command: |
            curl -H "Circle-Token: $CIRCLE_TOKEN" https://circleci.com/api/v2/pipeline/<< pipeline.id >>/workflow > workflows.json
            WORKFLOW_ID=$(jq -r '.items | map(select(.name == "<< parameters.workflow-name >>")) | .[0].id' workflows.json)
            echo "export WORKFLOW_ID='${WORKFLOW_ID}'" >> $BASH_ENV
      - run:
          name: Find Job ID for << parameters.job-name >>
          command: |
            curl -H "Circle-Token: $CIRCLE_TOKEN" "https://circleci.com/api/v2/workflow/${WORKFLOW_ID}/job" > jobs.json
            APPROVAL_JOB_ID=$(jq -r '.items | map(select(.name == "<< parameters.job-name >>" and .type == "approval")) | .[0].id' jobs.json)
            echo "export APPROVAL_JOB_ID='${APPROVAL_JOB_ID}'" >> $BASH_ENV
      - run:
          name: Approve job
          command: |
            curl -X POST -H "Circle-Token: $CIRCLE_TOKEN" "https://circleci.com/api/v2/workflow/${WORKFLOW_ID}/approve/${APPROVAL_JOB_ID}" | jq .

In the spirit of sharing, I have created a CircleCI Orb that codifies the above job for your convenience.

https://circleci.com/developer/orbs/orb/kelvintaywl/control-flow

I hope this article and the Orb will be useful. Keep on building, folks!

#circleci #cicd #workflow

buy Kelvin a cup of coffee ☕