Making a multi-track recorder in Flash part 2

On Tuesday night I did a last minute ad hoc presentation to FlashBrighton to share the experiments I’ve done with the microphone capabilities of FlashPlayer 10.1. You should be able to see the recording at live.flashbrighton.org (if it’s not there, bear with us while we try to get it working). Click “media” to see the available videos.

I had a lot of fun and there were actually 85 people watching the live stream and heckling me (including Lee Brimelow and John “Flash on the Beach” Davey), which I really enjoyed, despite my feigned irritation. 🙂

I explained exactly how audio works, what it means when we see a wave form, and how that gets turned into audio that we can hear. And then I showed that a waveform gets all jumbled up when you mix sounds, but our brain can still separate it all.

Then I showed how simple it was to make a sine wave tuned to concert A, 440 Hz. And then we mixed in the C above it to make a two note chord. Here’s the code for that :

 
package
{
	import flash.display.Sprite;
	import flash.events.SampleDataEvent;
	import flash.media.Sound;
	import flash.utils.ByteArray;
	
	public class SimpleTone extends Sprite
	{
		
		public var counter : int = 0; 
		
		public function SimpleTone()
		{
			super();
			
			var sound : Sound = new Sound(); 
			sound.addEventListener(SampleDataEvent.SAMPLE_DATA, getBytes); 
			sound.play(); 
		
		}
		
	
		public function getBytes(e : SampleDataEvent) : void
		{
			// we'll buffer 4096 samples into the audio data
			
			for(var i : int = 0; i<4096; i++)
			{
				counter ++; 
				
				// some crazy maths here! But it's simple when we break it down... 
				// We want the sine wave to oscillate 440 times a second (concert A) and Math.sin
				// completes a full wave in 2 x PI. So from 0 to 1 second we want to give 
				// Math.sin a value from 0 and 440 x 2 x PI. 
				//
				// And as there are 44100 samples in one second, we divide the counter by that number. 
				//
				// Easy! ( or at least it would be if I could explain it better... )
				// 
				
				var sample : Number = Math.sin(counter/44100 * 440* 2 * Math.PI) *0.5; 
				
				// now add the wave for the C above A. 261.626 is the frequency of a middle C
				// but we want the octave above that, so we multiply it by two. By adding this to 
				// our sample value, we're combining the two notes. 
				
				sample+= Math.sin(counter/44100 * 261.626 *2 *2 * Math.PI) * 0.5; 
				
				// it's a stereo sound so we have to add the sample once for the left and once
				// for the right. 
				e.data.writeFloat(sample); 
				e.data.writeFloat(sample);
				
			}
			
		}
	
	}
}

So then we went on to look at how we get recorded bytes from the microphone, here’s a simple microphone recorder and looper, hit the mouse to loop what you recorded.
[sorry, flash content no longer available]

Don’t forget that you need FlashPlayer 10.1 for it to work!

I’ve also added some code that toggles playback so it doesn’t get too annoying if you’re reading this post 🙂 But here’s the simple version of the code :

 

package
{
	import flash.display.Sprite;
	import flash.events.Event;
	import flash.events.MouseEvent;
	import flash.events.SampleDataEvent;
	import flash.media.Microphone;
	import flash.media.Sound;
	import flash.media.SoundChannel;
	import flash.utils.ByteArray;
	
	public class SimpleMicRecorder extends Sprite
	{
		
		public var mic : Microphone;
		public var sound : Sound;
		public var soundChannel : SoundChannel; 
		public var micBytes : ByteArray; 
		
		public function SimpleMicRecorder()
		{
			super();
			
			// set up the microphone
			mic = Microphone.getMicrophone(); 
			mic.setSilenceLevel(0); 
			mic.rate = 44;
		
			// by adding the event listener, we're starting recording
			mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micBytesReceived); 
			
			// and we'll make a byte array to store our recorded bytes into
			micBytes = new ByteArray(); 

			// and when we hit the mouse, we'll stop recording and start 
			// looping what we recorded. 
			stage.addEventListener(MouseEvent.MOUSE_DOWN, stopRecording); 
		
		}
		
		
		public function micBytesReceived(e : SampleDataEvent) : void
		{
			// append the data for the sound we've just recorded onto the end
			// of the byte array we're storing it all into
			micBytes.writeBytes(e.data); 
		}
		
		
		public function stopRecording(e : MouseEvent) : void
		{
			// removing the event listener from the mic in effect
			// stops the recording
			mic.removeEventListener(SampleDataEvent.SAMPLE_DATA, micBytesReceived); 
			
			// and set the byte array position at the beginning so 
			// that when we start reading data from it we'll get 
			// the start of the recording
			micBytes.position = 0; 
			
			// so, make the sound object and start playing it!
			sound = new Sound(); 
			sound.addEventListener(SampleDataEvent.SAMPLE_DATA, getBytes); 
			soundChannel = sound.play(); 
			
			stage.removeEventListener(MouseEvent.MOUSE_DOWN, stopRecording); 
		}
		
		public function getBytes(e : SampleDataEvent) : void
		{
			// we wanna give the sound object enough samples to play. 
			// I find that it borks if you give it too many or too 
			// few samples. 4096 seems a mid ground. 
			
			for(var i : int = 0; i<4096; i++)
			{
				// if we've run out of recorded bytes, move the pointer
				// on the byte array back to the beginning so that we're
				// now reading the bytes over again. In other words, 
				// the sound is looping. 
				if(micBytes.bytesAvailable == 0) micBytes.position = 0; 
				
				// read the next sample out of the mic byte array
				var sample : Number = micBytes.readFloat(); 
				// and write it twice into the sound data, once
				// for the left and once for the right. 
				e.data.writeFloat(sample);
				e.data.writeFloat(sample);
			}
		}
			
		
	}
}

You may notice in the video that I don’t quite get this to work. This is because I didn’t know what I was going to be doing until 6pm that night 🙂 And also because I was trying to play back bytes of data while I was still recording. This causes the pointer in the byte array to jump around and it gives very unexpected results!

So, next up I started talking about the challenges in making a multi-track recorder. The main one is that we have no accurate idea where the bytes of microphone data happen in actual time. And we also don’t really have a very accurate way to know when an audio file is playing.

Why is this a problem? Well imagine we’re making a multi-track recorder. We record one track, and then loop it. We record a new track over the top of it. The new track needs to be placed over the old track so that they’re perfectly synched together. If we don’t know when the microphone data happened in relation to the recorded audio then we can’t possibly sync them up!

This did my head in for ages. I tried different methods of finding out timing information from the sounds and the microphone data. I found some useful information :

SoundChannel.position – this is useful. As I posted earlier, it returns the position in milliseconds of the sound. For dynamically generated sounds (as in this example), I believe this is the time that the sound has been playing for.

SampleDataEvent.position – this is the event that is passed through when you’re either playing a dynamically generated sound or when you’re recording microphone input. In an earlier post I explained that this was in samples. Now I’ve checked this again, I can’t actually figure out what this is measured in. All I know is that to convert it to mils, I’m dividing by 8, which makes no sense to me. Anyone got any ideas about this? It is actually measured in samples. But I was confused because the default sample rate for the mic seemed to be 11K, not 44.1 as I assumed.

So here’s an example where I’m starting a timer, starting the mic recording, then starting the sound, and each line represents each one of those three. The red line is the timer, and that stays fixed. The next line is the mic position, extracted out from the mic’s SampleDataEvent, and divided by the sample rate to get mils, and the last is the SoundChannel position.

[sorry, flash content no longer available]

See how the three jump around? Each pixel represents a millisecond, and as even a misalignment of 30-50ms would sound weird, we clearly haven’t got the accuracy that we need. So how do you fix that? Well I worked out a system, it’s a bit hacky but it seems to kinda work. And I’ll write it up in the next post (or if you can’t wait, watch the recording from FlashBrighton).