hammertoe

Writing simple trading bots, or algorithms, to trade a cryptocurrency, commodity, or stock is pretty simple, right? You just need to buy low and sell high... easy, right?

Well as anyone who has attempted to do this will tell you, it's not that simple, as there are a myriad of complexities. In this post I'll talk about just one aspect, and that is 'hyperparameter tuning'. I'm going to slightly abuse the term hyperparameter here for this example. Typically, a hyperparameter is term used in machine learning to describe a 'meta parameter'. That is, not the parameters that the machine learning algorithm itself is learning, but the parameters about the learning as a whole.

An analogy: what books I decide to read at university in order to learn might be a parameter, but what university I go to in order to do that might be a hyperparameter.

In this post I'm talking about using a python library called hyperopt to tune the parameters of a simple trading algorithm. So we are actually tuning parameters, not hyperparameters, but the library doesn't care. Hopefully this will become clear below. Just think of it as trying to tune some parameters.

Lets say we devise a very simple trading strategy to instruct some software to automatically trade a currency (or cryptocurrency, stock, commodity, etc):

Buy when the price goes above the 21-period moving average, sell when it goes below it.

Pretty simple, huh? The aim is to try and detect some kind of trend in the movement.

In that algorithm, “21” is a (hyper)parameter. It could be 18, or 53, or 1, or 10,000. How do we choose the best one to ensure that our algorithm is profitable? Too small and the algorithm will trade too much and likely lose money in fees. Too big and we'll only trade once in a blue moon and by then the price will have already moved quite a lot.

What does the problem look like?

Let's take some guesses and plot them out and see what they look like. In the charts below I'm plotting the price of USD/JPY and a 5 period moving average. The green triangles indicate where the price rises above the moving average and we should buy. The red ones when we go below and should sell.

Quite noisy. Lots of trades going on. And we will be charged a commission on each trade, so will lose a small amount each trade. Let's look at a few more:

As you might be able to see we do really badly on some of them. Look at the last one. We buy in mid-February at around 110.2 Yen and then sell in mid-June at a lower price of around 107.5 Yen... not what we want to do!

What can we do?

So what is the best number to use? What is the best figure for a moving average that we should use? That is where the parameter tuning comes in.

We could just try every number. Computers are fast, right? Try every number between say 4 and 200 and see what works best. That is only 196 possible outcomes. This is known as a 'brute force' approach. That will take a computer less than a second to work out.

But what if our strategy is more complex? What if we have several parameters we need to tune simultaneously? Rate of upward trend, rate of downward trend, stop loss position, etc? We could quickly end up with hundreds of thousands or even millions of combinations to try out. And what if we are wanting to test on several years of data? Maybe using 5-minute intervals, not daily intervals? What if each attempt takes longer as we are testing out our SuperFancyUltimateMoneyMaker2000 strategy?

Brute force won't cut it. We could be waiting for hours, days or weeks for a computer to try all possible combinations.

So what if it could do something more clever? What if it could try some random combinations and then look to see if they give good results or not, and if they do, then try other values 'near' those other good ones.

This is what Hyperopt lets us do. It uses an algorithm called Tree-structured Parzen Estimator (TPE) to more intelligently 'search' the space of all possible combinations to find the best ones. The end result is something that could take a whole day to search all best combinations by the brute for approach can now take mere minutes.

And what is great with Hyperopt is that it is really simple to use it. You need to define two things:

1. Your function that you want it to run. In this case a function that simulates trading as detailed above and takes one or more parameters you want to optimise.
2. The parameter space.

Below I'll dive into the actual code and the results of the optimisation we did. If you are not a Coil subscriber, now will be a good time to subscribe ;)

Header photo by Mikael Kristenson on Unsplash

So following on from my last post about trying to generate cartoon ducks using AI — and accidentally producing something quite Warhol-ish, I decided to try and generate a new profile pic for myself on our intranet at work. A colleague of mine said that my current pic makes me look Mr Noodle from sesame street. No. You can't see the pic. But maybe my beard was a bit too unruly at the time, and maybe the wall behind me was a bit too bright and primary-colourish.

This is a post in my series on machine learning and artificial intelligence. You can find more posts on this topic at the main index.

[](https://coil.com/p/hammertoe/Blog-Machine-Learning-and-Artificial-Intelligence/1xVFVh1yj)

So could I use the same technique of Generative Adversarial Networks (GANs) to produce a new image of “me”?

Let's recap how these networks work with a little analogy:

“So Sir, can you describe to us the person who robbed you of your wallet?”

“Yes officer, he was male, early forties, caucasian, 5'9”, short brown hair, glasses, a beard and moustache”

[sketch artist works furiously]

“Like this?”

“No, the glasses were thinner, wire-framed type”

[sketch artist draws a new drawing with different glasses]

“Like this?”

“Yeah... maybe smaller nose”

[sketch artist draw a new drawing with smaller nose]

We have two neural networks, one, the generator (the sketch artist) and a second one, the discriminator (me). The first one is creating new images and the second one is trying to critique them. If the critic can't tell the difference between a 'fake' and a 'real' image, then the generator (sketch artist) has learned how to produce good likenesses of the subject.

So first, I needed a whole load of real images for the generator to feed to the discriminator in amongst its 'fake' ones to see if it could tell the difference.

Luckily Apple iPhones already have some machine learning in them to identify and categorise people. So I can easily copy 300 pictures from the last 5 years of myself from the phone to my desktop computer for processing.

I then opened them all up in Preview and very quickly and roughly cropped them to just have my face in. I discarded those that I was wearing sunglasses, or at a very odd angle to the camera.

I then fed those images into the GAN from before. And out of the gaussian noise, started to emerge a somewhat recognisable me...

Further...

So there, are definitely some likenesses there, but still in most of them I look like some apparition from a horror film.

I realised, that the images all being slightly different crops and orientations was giving the GAN a hard time. Bearing in mind this is a fairly simple network and we are far from the state of the art.

So I realised that I could use another bit of machine learning to pre-process the images. I could use a computer vision library to detect my facial features and then rotate, shift and scale the image such that at least my eyes were in the same place in each photo.

If you want to know the full technical details and code on this, I've written a separate (subscribers only) post on Using OpenCV2 to Align Images for DCGAN.

But the end result was pretty cool... and quite spooky. So here is an animated gif showing a few of the input images:

Notice how my face is slightly different size, and eyes moving about in different locations? Compare that to the aligned images:

Pretty eerie, right? Those are the same images as above, but the pre-processing AI has calculated the location of my eyes, and transformed the image such that my eyes are in the same location in each image.

So lets feed that back into the GAN and see how we do...

Not bad! There are certainly some ones there were it looks like I've been hit in the face with a shovel a few times... but actually overall it has done pretty well.

Again, what is pretty amazing is that none of these images ever existed. They are not distorted photos, but they are the result of a machine learning algorithm learning what my facial features look like and creating an entirely new and unique image of me.

So to recap, in the end there were three entirely separate machine learning / AI algorithms used in this process:

1. The machine learning on my iPhone that had analysed all my photos and clustered the ones of the same same people together, such that I could easily find photos of me
2. The OpenCV script used to pre-process the images that located my facial features in the photos and the rotated, scaled, and shifted the image such that my eyes were in the same place on each image
3. The Generative Adversarial Network (GAN) that was used to generated entirely new images from the existing ones.

For coil subscribers, there is another animation showing the learning as it progressed.

So, firstly, thanks to Alloy for the Magritte reference for the title of this post.

[](https://coil.com/p/hammertoe/Blog-Machine-Learning-and-Artificial-Intelligence/1xVFVh1yj)

This is a post in my series on machine learning and artificial intelligence. You can find more posts on this topic at the main index.

Yesterday, a friend on Twitter, Munch, put out an art challenge:

Well, I'm rubbish at art. But for a bit of fun, decided that rather than showcase my rubbish art, could I instead showcase a computer's attempt at art? What could go wrong? Can I teach a computer to draw a duck and fill in the blank template above?

So, introducing to you the concept of Generative Adversarial Networks... or GANs for short. Gen...what? These are a type of machine learning neural network that attempt to generate new data (generative) from learning from a series of samples, by trying to out-wit a second neural network (adversarial).

So how does it work? Let's imagine you and I play a game. Your job is to show me a series of James Bond films some of which are real, and some of which are ones that you have produced yourself. My job is to try and guess which of the films is real, and which is fake. Got it so far?

So firstly, you watch all 26 James Bond films and try to distil down the essence of a James Bond film. What features make it a James Bond film if you had to describe it to someone?

• Athletic male lead, who is a British secret service agent
• Lots of gadgets like X-ray glasses, and camera tape recorders
• Car chases... cars with guns in them etc
• Esoteric villains
• A sexy adversary
• Great music
• etc

Using that list of 'features' of a Bond film, you now produce your own Bond film... you might be good at it you might be bad at it. To start with you are going to be pretty lousy at it on your first attempt.

My job is to look at the film you produce and decide if it is a real Bond film or fake. If it is very easy for me to guess a real one from a fake and I get them all right, then that means you are doing a poor job at making fakes.

The goal here, is for you to keep trying to make fakes to fool me. And my job is to keep feeding back to you how well I can tell the real from the fake. If I can't tell the difference from a real Bond film and a fake one, then you have done a very good job in producing a new, unseen, fake Bond film.

In terms of a machine learning architecture, it looks like this:

source: https://www.slideshare.net/xavigiro/deep-learning-for-computer-vision-generative-models-and-adversarial-training-upc-2016

You have one branch at the top (yellow) feeding in real Bond films, and you have a generator at the bottom (blue) creating fakes ones (your job). And you have me, the discriminator (red), trying to tell if they are real or fake. My error rate in judging them, loss, is fed back to the start for the next iteration. Based on that loss, you can determine which features were important or not. ie. if you put a dragon in your Bond film and I feed back that clearly this is not a Bond film you can infer that dragons are not a legitimate element of a Bond film.

Show me the ducks!

So, I needed:

• a working GAN algorithm
• some ducks

For the algorithm, I found one on Github (a code sharing/collaboration site) that I could use for this task. This one had been used originally to try and generate new Simpson's characters, so might be good, as already tuned to cartoons:

I loaded the code into Google Colab, which is another collaborative tool that allows you to run python code on a cluster of computer run by Google. This is helpful as GANs take quite a lot of computing power, and means I could run it faster than I could on my computer at home.

I also needed a bunch of 'real' images of duck drawings for it to use. I search Google for “draw a duck share your art” which is the meme this was from, and downloaded a whole bunch of them:

I uploaded them to where my code could load them and then set the algorithm to run. My first attempt was for 300 'epochs', that is there were 300 iterations of trying to generate duck images and trying to fool the disciminator.

The first attempts you can't see anything duck-like at all, as the generator has started with random data and still not worked out what 'features' the output images should have. Here is 5 samples of 'ducks' from epoch 24:

Not very exciting, huh? Just grey blobs...

As it progresses, it starts to try various colours... and we end up with a nice tweed sort-of colour coming up. Great for cushions perhaps, but still not a duck. Epoch 100:

By the time we get to epoch 250, we are now getting some slight duck-like features show in the images. You can spot the yellow feet and beak emerging:

But by the time we get to epoch 300, it's all gone wrong again:

What happened?! Well it turns out it looks like the algorithm kinda overshot the mark. There is actually a graph produced of the learning losses of both the generator and the discriminator:

You can see that they were converging until around epoch 250, then the generator kinda went a bit nuts.

So, I decided to run it again, reducing the 'learning rate' of the algorithm. This means it will learn a bit slower, but hopefully won't overshoot as much. I left it running overnight for 600 epochs to see how it got on. The graph of the learning looks a lot more positive this time. As you can see by the time it got to the end it was still pretty stable. It looks like it still has a bit of a 'wobble' around 300, but then seems to stabilise again.

So... the moment of truth... let's see what our ducks looked like!

Here is a sample of them, from near the end at iteration 589:

Whilst I don't think I will be winning any awards with these ducks, I think it has certainly managed to get along the right tracks!

I think due to the high variance of input duck images, it had a hard time trying to work out what a duck should look like. With more samples, and perhaps more iterations it might do better. There are also a number of parameters to the algorithm that can be tuned to try and get better results. But for a quick un-tuned attempt, I'm pretty impressed.

And for Coil subscribers below you can see a really cool animation of the 'evolution' of the algorithm learning what a duck is :)

Part II of this adventure continues with: Ceci n'est pas un Matt – Machine Learning and Generative Adversarial Networks – Part II

In this series I am going to be talking about machine learning and artificial intelligence. I will be covering three main branches of machine learning: supervised learning, unsupervised learning, and reinforcement learning. In this post I'll be talking about reinforcement learning, as it is one of the ones I find the most fun.

This is a post in my series on machine learning and artificial intelligence. You can find more posts on this topic at the main index.

If you haven't done already, and need a primer, go and read my introduction to machine learning that I wrote previously: What is Machine Learning / Artificial Intelligence?

So, here we go. Who has one of these at home?

No, not an Ewok... it is in fact a dog, Violet, my pug/bichon cross. If you have a dog or are a parent, you probably have already come across reinforcement learning. It is the act of learning via a reward. Reinforcing good behaviour with a treat. Perhaps discouraging unwanted behaviour via a punishment.

You tell your dog to sit. It has no idea what the word 'sit' means. It looks at you. You give it a hint, maybe, holding a treat above it's head so it sits down to look up. And you give it the treat. You repeat this over and over. And the dog learns to associate the command 'sit' with the desired outcome.

Reinforcement learning in computers is pretty much exactly the same. It is an iterative process by which an algorithm 'learns' desired behaviour by way of being given a 'reward'.

This process is shown above as a state diagram. We have an agent (the dog), that performs an action (sits), in an environment (your house), which results in a new state (a sitting dog) and a reward (tasty treat).

What is so great about reinforcement learning is that you don't describe how the result should be achieved, just reward the desired outcome. ie, you don't tell your dog to bend it's rear legs to sit its backside on the ground. You just reward it when it sits down.

Let's move away from pets, and show this in the context of a computer learning to play a simple game:

Do you remember the Atari game “Breakout”? In the game you control a small horizontal “paddle” at the bottom of the screen. You have a joystick or controller and can press “left” or “right” to move the paddle. A ball bounces around the screen and your job is to stop the ball falling out the bottom of the screen. A bit like pinball. You have to bounce it off the coloured blocks and the top and try to destroy them all. You get a score for each block you destroy.

We can use reinforcement learning to teach a computer how to play this game. We have two possible actions: left and right, we have a state which is the grid of coloured pixels that make up the screen, and we have a reward, the high score. So the job of the algorithm is to look at the pixels (the image on the screen), move the paddle left or right, and observe the score as a reward.

At first the algorithm will do very badly. It has no idea what it is doing (just like a human player for the first few seconds playing). IT will randomly move the paddle left and right. It has no idea what the coloured pixels on the screen represent, or even what the ball is.

But with each successive iteration it learns a little bit. In most cases RL algorithms use a neural network to process the images. I'll describe these in a later post, but they are effectively a model of a human brain and the neurons in it.

Above is a graph showing the score after each game played of breakout for various RL algorithms. Each coloured line is a different algorithm, and each one learns in a slightly different way. The axis along the bottom is in millions of steps, so this graph covers two million steps (iterations) in total. As you can see, at the start, they all get a pretty low score. Then over time, they learn. Some faster than others. For example, PPO doesn't get a very high score overall (around 150). Others, e.g. C51 got scores over 500.

At the end, the algorithm has learned that it needs to position the paddle to where the ball is heading in order to bounce the ball back up to the blocks. It has learned to play the game.

What else can we use reinforcement learning algorithms for?