cenne

08 Aug 2021 (uploaded 2021-09-01)

Setting up the environment.

# Note: TBD. Link to the project's documentation for now. 

Here's how the contributor docs of the ironic project recommend you create an environment.

# create separate env for ironic work (do once)
virtualenv -p python3 $HOME/envs/ironic-env

# activate the environment (whenever working on ironic)
source $HOME/envs/ironic-env/bin/activate

# Close the environment (when done working or switching to something else)
deactivate

With the ironic-env active, installing necessary packages is as simple as usual pip install.

# for running unit-tests and code-formatting checks
pip install tox

# other project requirements
pip install -r ironic/requirements.txt
# Note: In the above command just replace `ironic` with the path to your ironic project directory

(source) This is very useful. I'll add my own tips for streamlining the work flow below soon.

Tip-1: Use an alias

Doing source ~/envs/ironic-env/bin/activate every time is pretty inconvenient. Especially when you have multiple project environments in ~/envs. Add a bash alias to get to your environment quicker, like so:

echo "alias ironic-env='source ~/envs/ironic-env/bin/activate'" >> ~/.bashrc

Next time, when you open the terminal, typing just ironic-env will activate your environment.

Tests (tox) shortcuts.

So, you just made a little (or big) modification to a file. The first thing you should do next is to write some accompanying tests. Usually this is to see * i) if your change actually works as you intended, * ii) to test how it behaves under 'weird' scenarios. You want to make sure that errors are handled gracefully (or failed hard at, depending on design).

Say you made some changes at ./conductor/node.py For the above file, we would write our unit tests in ./tests/unit/conductor/test_node.py We use a project's directory like structure for tests, just one level under the tests directory.

Okay done, you have your tests, and you have the modification you made. Now how to test? We'll use tox for that. We installed it earlier in the above section.

Basic unit tests + specifying smaller test runs

Basic way to do unit tests is:

tox -e py3

You might have seen the above in the contributor docs. This will run all the tests it finds under unit tests directory. This takes a looong time. You can narrow down the tests to save time. For example

tox -e -- ironic.tests.unit.conductor 
tox -e -- ironic.tests.unit.conductor.test_node

How did we get this magic incantation? Just replace '/' in the directory structure with '.' You can also go a level deep into files, to test specific classes.

Just like how you would do imports in python.

Code Formatting

Openstack/ironic requires your code to conform to formatting guidelines described in pep8. To see if you are doing so and to get suggestions on how to fix discrepancies, run the pep8 test.

tox -e pep8

Re-run only failing tests

Sometimes you may find yourself repeating running tests, while making small changes trying to fix your code that's failing. At such times, you may find the --failing option useful.

tox -e -- --failing

This reruns only those tests that failed in the previous test-run. You can keep repeating it until you've fixed all those bugs. Remember to do a full test run after to make sure you've not introduced any new bugs.

Since the --failing option only runs unit tests that failed previously, if you introduce a bug, you won't know about it until you re-run the previously successful tests again.

First, there were computers. A big bad computer was called a server. It could chomp down on calculations and serve many people at once.

But a computer needs an owner, a manager who tells who to serve, what to do. And few could run it to its full potential.

Some like Marvin would languish depressed, doing meagre tasks for big-name figures, Others would just do their work and play with bugs when they were bored. Also, computers didn't come for free – the bigger you had, the costlier they were, and not to even mention the cost of electricity incurred to feed, and their regular individual maintenance.

Someone thought, 'what if we could break this computer's mind into many pieces?' – call them virtual machines as opposed to, you know, the real one sitting in front of you and who you've got accustomed to. That way we could hand each of these virtual machines to different people or to run totally independent projects. Hand them virtually of course, you would still have the machine in front of you and in your control, but you could rent mindspace to people far away, or to your friends or people working for you.

This idea turned out great, most people knew how to use a computer, but not necessarily a bigger computer well. By breaking it into smaller virtual machines with power that people actually needed, efficiency improved. Also, each virtual machine (VM) user had less to worry about other users knowing the secrets they shared with their VM. Since it was not exactly one machine everyone was using, but different self-contained computing environments or mind spaces. So even if your neighbouring user broke their VM, you'd be least bothered – yours would continue to work undisturbed.

And so, this situation was working well, still is as of year 2021. Many people came up with their solutions for doing this virtual machine business. You had commercial vendors like AWS and Azure. On the other hand, there were also open source initiatives like OpenStack.

One word to introduce here is 'The Cloud'. It is essentially just a bunch of virtual machines with accompanying virtual network, virtual storage and other solutions. If you have many real machines, and you turn them into loads of virtual machines and enable people to conveniently use them from far away, you have a 'Cloud'. Here's how OpenStack describes itself:

What is OpenStack? OpenStack is a cloud operating system that controls large pools of compute, storage, and networking resources throughout a datacenter, all managed through a dashboard that gives administrators control while empowering their users to provision resources through a web interface.

Now, continuing the story, some people decided they did not like these 'virtual machines'. They needed a real full machine all for themselves. They had their reasons, say requiring increased security or power, or just wanting more control. And so they tasked this cloud orchestrator originally built to generate and manage virtual machines to hand them an interface to the full solid bare-metal machine instead. Now, isn't that ironic? Yes, indeed, that's Ironic! Literally :)

Ironic (OpenStack's Bare Metal Provisioning Program) is the project I am working on as an intern. I shall give you a walk through its internals in a subsequent post.

Caution! Note: This piece has ample contribution from author's imagination and common folklore. As such, it is not meant as a history lesson. Think of it like a grandma explaining ironic to a five-year-old. Which in itself self is ironic because the author feels like a 5y old in comparison to assured readers.

For some history see – https://en.wikipedia.org/wiki/Cloud_computing#History

Okay, so today I'm going to tell you a little story through code. This is a partial story about session and my encounters with this mysterious entity. Bear with me, it's going to be a ride (probably a trippy one).

Prerequisite knowledge: – Python (intro level), – HTTP request methods

I first encountered this object, not just under that generic name, but as a mysterious entity that did other things. Those who received this session, said it was of a particular type (or kind) ~keystoneauth1.adapter.Adapter That description seemed just as cryptic. But it did give a hint.

Here's a sample of our first meet.

    #location: openstacksdk/openstack/baremetal/v1/node.py - Node/

        def commit(self, session, *args, **kwargs):
        """Commit the state of the instance to the remote resource.

        :param session: The session to use for making this request.
        :type session: :class:`~keystoneauth1.adapter.Adapter`

        :return: This :class:`Node` instance.
        """

Hmm. Now what did that tell me? That there is this function called commit. And it receives this session gladly as a parameter (required it, in fact). commit says session is of some particular type.

Now, I had not encountered this type before. But from the name I figured this session was someone that authenticated you and then allowed you to do things with it. Sort of like a magic wand. Perhaps it first identifies you, and then it can be used to make this happen, make others (maybe a server) do things for you. Also, just like a wand, it can be passed around. So it must have some memory or state of who or how it was charmed. This is all just guess work though, from what we see here.

With not much of a prior introduction to it and not knowing where it came from, I wandered around this place that is the openstacksdk trying to find more about it. You see, I was supposed to implement a feature at this place and to do so, I needed to use this session to get some things done. I wanted to get to know all about this session first Who it was, what does it do. But alas, my search failed. I didn't find a more detailed description nearby. Tracing back, following it, through the signs it left, I saw it invoked like this.

from keystoneauth1 import adapter

from keystoneauth1 import session as ks_session

It was always summoned, not a native of this place (openstacksdk) but brought in from outside. So I didn't find anyone having detailed information about this session. It was one mysterious keystone. I was dismayed, I had to use it. How would I, if I didn't know who it was, or know how to talk to it?

I was saddened, I was stuck. I was scared of not progressing, and almost fell into a rut.

My mentor fortunately noticed, and helped me out, they pointed me at examples, encouraging me, to figure it out.

Here is one such example

    def list_vifs(self, session):
        """List IDs of VIFs attached to the node.

        The exact form of the VIF ID depends on the network interface used by
        the node. In the most common case it is a Network service port
        (NOT a Bare Metal port) ID.

        :param session: The session to use for making this request.
        :type session: :class:`~keystoneauth1.adapter.Adapter`
        :return: List of VIF IDs as strings.
        :raises: :exc:`~openstack.exceptions.NotSupported` if the server
            does not support the VIF API.
        """
        session = self._get_session(session)
        version = self._assert_microversion_for(
            session, 'fetch', _common.VIF_VERSION,
            error_message=("Cannot use VIF attachment API"))

        request = self._prepare_request(requires_id=True)
        request.url = utils.urljoin(request.url, 'vifs')
        response = session.get(
            request.url, headers=request.headers, microversion=version)

        msg = ("Failed to list VIFs attached to bare metal node {node}"
               .format(node=self.id))
        exceptions.raise_from_response(response, error_message=msg)
        return [vif['id'] for vif in response.json()['vifs']]

Looking close, ignoring the rest, we can focus on the session object and see how it behaves, try to glean how it's used.

session = self._get_session(session)
# Just by looking we can guess, perhaps this session is being refreshed to it's current state, or perhaps minorly transformed from a generic form into a form we can use. 
# And indeed looking at it's source, indicates the latter


response = session.get(
            request.url, headers=request.headers, microversion=version)

# Ignore the microversion for now. But if you know basic http, looks like what's happening is a get request is being initiated at the url, and the returned response is being saved.
# For some magicians it would be natural to just wave this s̶p̶e̶l̶l̶ fuction of session at any url and hope it works.
# Some even more daring would probably even try `session.put( ... ) ` or session.delete( ... ) and expect it to work*. 
# Some would be too distracted by this strange microversion and hesitate to touch the function until they know more, fearing side effects. (like I was at first :p)

# * (GET, PUT, POST, DELETE being some classic HTTP request methods)

Browsing through the code, I did find some other incantations. Like:

        response = session.put(
            request.url, json=body,
            headers=request.headers, microversion=version,
            retriable_status_codes=_common.RETRIABLE_STATUS_CODES)

        response = session.post(
            request.url, json=body,
            headers=request.headers, microversion=version,
            retriable_status_codes=retriable_status_codes)

        response = session.delete(
            request.url, headers=request.headers, microversion=version,
            retriable_status_codes=_common.RETRIABLE_STATUS_CODES)

I tried these incantations myself (skipping the extra parameters). And it did seem to work. So it does seem like an interface to HTTP methods. This session wand allows you to call HTTP methods. You can pass it the url, headers and json body in case of post/put request and it seemingly does the job. How it works is magic (a black box). What else it does, is also a mystery.

For my purpose, this was all I needed from our mysterious session. And so we parted ways. Until we met again, another place.

I hadn't seen fine details of our session. Didn't know where exactly they were from. Hadn't seen how session was created, haven't read about their usage or detailed description in a document. And that felt blinding.

But, just by observing someone's behaviour, seeing how others call them, how they react to those calls, you can sometimes glean enough about an entity (Perhaps even a person).. Enough to work with them.

Who they are may remain a mystery, and you may never get to know, as much as you'd like. But's that's okay. You don't have to always.