Rockets and Music
Fifty one years ago the famous words of Neil Armstrong were heard around the world:
one small step for man, one giant leap for mankind
Man had landed on the moon. The Apollo 11 lunar module had successfully descended to the surface of a lump of rock a quarter of a million miles away.
Onboard the lunar descender was a guidance computer that calculated the descent trajectory to the landing site. One of the key pieces of information it needed was the position of the lunar module itself in space. This position information was taken from a series of four Doppler radar stations on earth. The problem was, how to tell that the readings were accurate? These readings were used to adjust the onboard systems. If either reading was too far out then the mission would have to be aborted.
Enter Rudolf Kálmán, a Hungarian mathematician. He had created an algorithm that could take a series of measurements from multiple sensors that might contain statistical inaccuracies and combine them to give a single, more accurate reading. On a visit to the NASA Ames Research Centre an aerospace engineer Stanley F. Schmidt realised that this algorithm, the Kalman Filter, could be used for the Apollo navigation computer.
Since then, Kalman Filters have been used in guidance systems from nuclear ballistic missile submarines to the navigation systems of the Tomohawk cruise missile. They are also used in the navigation systems of the craft that dock at the International Space Station.
So what has this got to do with music?
I have been working on the synchronisation algorithm for Choirless, a website that allows people to sing and play music remotely. Each person plays or sings their part, and Choirless combines them all together and gets them all in time with each other.
https://cinnamon.video/watch?v=444770063212348429
The magic behind the scenes analyses each audio waveform and extracts from it certain 'features' of the audio. These features include the rate of change of frequency (spectral flux), the rate of change of volume (crest factor) and the actual dominant musical note (chroma). These are used to try and measure how similar the audio files are at different offsets so it can determine the best alignment such that everyone is in time.
We take many readings from each of the features above, taking multiple attempts at calculating the synchronisation of the muscial piece with the reference piece.
In the chart above each of the faint red, green and blue lines represent a single attempt to synchronize the music. The horizontal access represents the offset from the reference piece. The vertical axis represents a measure how different the two pieces of audio are at that offset. So the idea is to find the lowest point (the least difference). In the chart above, the offset which fits best, is at 81 milliseconds. That means if we trim 81 milliseconds off the start of the audio file it will be in perfect time with the reference piece. But if we have multiple readings for the same thing, how do we work out which is the 'true' reading (the solid red, green and blue lines)?
A Kalman Filter.
So an algorithm developed sixty years ago and with its first major application in landing man on the moon, is now being used to synchronize music together to enable people from across the globe to sing together.
