Categories

# Making a multi-track recorder in Flash part 2

On Tuesday night I did a last minute ad hoc presentation to FlashBrighton to share the experiments I’ve done with the microphone capabilities of FlashPlayer 10.1. You should be able to see the recording at live.flashbrighton.org (if it’s not there, bear with us while we try to get it working). Click “media” to see the available videos.

I had a lot of fun and there were actually 85 people watching the live stream and heckling me (including Lee Brimelow and John “Flash on the Beach” Davey), which I really enjoyed, despite my feigned irritation. ðŸ™‚

I explained exactly how audio works, what it means when we see a wave form, and how that gets turned into audio that we can hear. And then I showed that a waveform gets all jumbled up when you mix sounds, but our brain can still separate it all.

Then I showed how simple it was to make a sine wave tuned to concert A, 440 Hz. And then we mixed in the C above it to make a two note chord. Here’s the code for that :

```
package
{
import flash.display.Sprite;
import flash.events.SampleDataEvent;
import flash.media.Sound;
import flash.utils.ByteArray;

public class SimpleTone extends Sprite
{

public var counter : int = 0;

public function SimpleTone()
{
super();

var sound : Sound = new Sound();
sound.play();

}

public function getBytes(e : SampleDataEvent) : void
{
// we'll buffer 4096 samples into the audio data

for(var i : int = 0; i<4096; i++)
{
counter ++;

// some crazy maths here! But it's simple when we break it down...
// We want the sine wave to oscillate 440 times a second (concert A) and Math.sin
// completes a full wave in 2 x PI. So from 0 to 1 second we want to give
// Math.sin a value from 0 and 440 x 2 x PI.
//
// And as there are 44100 samples in one second, we divide the counter by that number.
//
// Easy! ( or at least it would be if I could explain it better... )
//

var sample : Number = Math.sin(counter/44100 * 440* 2 * Math.PI) *0.5;

// now add the wave for the C above A. 261.626 is the frequency of a middle C
// but we want the octave above that, so we multiply it by two. By adding this to
// our sample value, we're combining the two notes.

sample+= Math.sin(counter/44100 * 261.626 *2 *2 * Math.PI) * 0.5;

// it's a stereo sound so we have to add the sample once for the left and once
// for the right.
e.data.writeFloat(sample);
e.data.writeFloat(sample);

}

}

}
}
```

So then we went on to look at how we get recorded bytes from the microphone, here’s a simple microphone recorder and looper, hit the mouse to loop what you recorded.
[sorry, flash content no longer available]

Don’t forget that you need FlashPlayer 10.1 for it to work!

I’ve also added some code that toggles playback so it doesn’t get too annoying if you’re reading this post ðŸ™‚ But here’s the simple version of the code :

```

package
{
import flash.display.Sprite;
import flash.events.Event;
import flash.events.MouseEvent;
import flash.events.SampleDataEvent;
import flash.media.Microphone;
import flash.media.Sound;
import flash.media.SoundChannel;
import flash.utils.ByteArray;

public class SimpleMicRecorder extends Sprite
{

public var mic : Microphone;
public var sound : Sound;
public var soundChannel : SoundChannel;
public var micBytes : ByteArray;

public function SimpleMicRecorder()
{
super();

// set up the microphone
mic = Microphone.getMicrophone();
mic.setSilenceLevel(0);
mic.rate = 44;

// by adding the event listener, we're starting recording

// and we'll make a byte array to store our recorded bytes into
micBytes = new ByteArray();

// and when we hit the mouse, we'll stop recording and start
// looping what we recorded.

}

public function micBytesReceived(e : SampleDataEvent) : void
{
// append the data for the sound we've just recorded onto the end
// of the byte array we're storing it all into
micBytes.writeBytes(e.data);
}

public function stopRecording(e : MouseEvent) : void
{
// removing the event listener from the mic in effect
// stops the recording

// and set the byte array position at the beginning so
// that when we start reading data from it we'll get
// the start of the recording
micBytes.position = 0;

// so, make the sound object and start playing it!
sound = new Sound();
soundChannel = sound.play();

stage.removeEventListener(MouseEvent.MOUSE_DOWN, stopRecording);
}

public function getBytes(e : SampleDataEvent) : void
{
// we wanna give the sound object enough samples to play.
// I find that it borks if you give it too many or too
// few samples. 4096 seems a mid ground.

for(var i : int = 0; i<4096; i++)
{
// if we've run out of recorded bytes, move the pointer
// on the byte array back to the beginning so that we're
// now reading the bytes over again. In other words,
// the sound is looping.
if(micBytes.bytesAvailable == 0) micBytes.position = 0;

// read the next sample out of the mic byte array
var sample : Number = micBytes.readFloat();
// and write it twice into the sound data, once
// for the left and once for the right.
e.data.writeFloat(sample);
e.data.writeFloat(sample);
}
}

}
}

```

You may notice in the video that I don’t quite get this to work. This is because I didn’t know what I was going to be doing until 6pm that night ðŸ™‚ And also because I was trying to play back bytes of data while I was still recording. This causes the pointer in the byte array to jump around and it gives very unexpected results!

So, next up I started talking about the challenges in making a multi-track recorder. The main one is that we have no accurate idea where the bytes of microphone data happen in actual time. And we also don’t really have a very accurate way to know when an audio file is playing.

Why is this a problem? Well imagine we’re making a multi-track recorder. We record one track, and then loop it. We record a new track over the top of it. The new track needs to be placed over the old track so that they’re perfectly synched together. If we don’t know when the microphone data happened in relation to the recorded audio then we can’t possibly sync them up!

This did my head in for ages. I tried different methods of finding out timing information from the sounds and the microphone data. I found some useful information :

SoundChannel.position – this is useful. As I posted earlier, it returns the position in milliseconds of the sound. For dynamically generated sounds (as in this example), I believe this is the time that the sound has been playing for.

SampleDataEvent.position – this is the event that is passed through when you’re either playing a dynamically generated sound or when you’re recording microphone input. In an earlier post I explained that this was in samples. Now I’ve checked this again, I can’t actually figure out what this is measured in. All I know is that to convert it to mils, I’m dividing by 8, which makes no sense to me. Anyone got any ideas about this? It is actually measured in samples. But I was confused because the default sample rate for the mic seemed to be 11K, not 44.1 as I assumed.

So here’s an example where I’m starting a timer, starting the mic recording, then starting the sound, and each line represents each one of those three. The red line is the timer, and that stays fixed. The next line is the mic position, extracted out from the mic’s SampleDataEvent, and divided by the sample rate to get mils, and the last is the SoundChannel position.

[sorry, flash content no longer available]

See how the three jump around? Each pixel represents a millisecond, and as even a misalignment of 30-50ms would sound weird, we clearly haven’t got the accuracy that we need. So how do you fix that? Well I worked out a system, it’s a bit hacky but it seems to kinda work. And I’ll write it up in the next post (or if you can’t wait, watch the recording from FlashBrighton).

## 15 replies on “Making a multi-track recorder in Flash part 2”

Fantastic Seb! I have had this idea of creating ‘live’ waveforms by adding harmonics via sliders for ages and this looks like a brilliant starting point. Where will it end? Digital effects? Fourier transformations? Brilliant.

Michaelsays:

That was a brilliant session at Flash Brighton (watched over the the broadcast!)…

The only thing I can think of for the dividing by 8 question that there are 8 bits in a byte and the byte array is actually storing bits… that’s probably nonsense, but I noticed the same thing when mucking around with sound in a byte array in the past – very odd!

Michaelsays:

Aha – a quick Google and Lee Brimelow has written about this here – http://theflashblog.com/?p=843 – describing how you could write 8 boolean values into a single byte…

yeah although clearly 8 is a significant number when it comes to bytes I can’t think of any reason that the microphone position would be the mils * 8. I’m making enquiries.

Oh and yes there are 8 bits in a byte, but my favourite is a nybble which is 4 bits. Half a byte. Cute. ðŸ™‚ Glad you enjoyed it Michael!

Michaelsays:

Hmmm, yes on reflection, there doesn’t seem to be a good reason does there… some arcane Adobe reasoning! However I’m just glad that it’s possible to ‘do’ sound properly in Flash now to be honest ðŸ™‚

Ryansays:

Great Presentation! Spent the past couple hours just tinkering around with it, I’ll spend a few more before I come it something half useful. Will you by chance be posting the multi-track example you were playing with at the end?

Hi Ryan! Glad you enjoyed it. The plan is to write a follow up article, but you know my track record on following through! But keep an eye out and feel free to hassle me again ðŸ™‚

Bart Geukenssays:

Hey Seb,

Really appriciate your work. But I have a question… Euhm, where can I find the source code from the live.flashbrighton.org session exactly? You said you were going to place it on your blog but I can’t find it? Even google can’t find it… Can you help me?

Bart

Hey Seb,

Enjoyed the live session along with everyone else on the call. I recently put together a video paying homage to a few of my favorite Flash “leaders”.

Hope you like it! ðŸ˜‰

iBrent

Slawsonsays:

Hey Seb,

So begins the hassling for the 3rd installment. looking forward to reading it.

Thanks

Slawson

thanks for the reminder Slawson. I’m at FlashBelt right now but I’m doing some more audio work so maybe I’ll get to it soon I hope! Please feel free to continue with these occasional reminders, just to keep me on track! ðŸ™‚

Very cool. I’d love to see that 3rd installment. There aren’t any multitrack recorders for Android phones yet, so I thought I’d see if there was a flash one that it could be used on the Android once flash 10.1 comes to it any day now.

No reason why it wouldn’t, right? I mean if there was one.

Lawriesays:

I know you’re probably winding down from your (by all accounts) incredible FOTB presentation, but I’d love to see the 3rd installment if you get the time. Cheers.

Hey Seb,

Great post, it’s nice to see people taking time to visually represent technical problems.

+1 for the hassling of the next instalment!

ðŸ™‚

Andysays:

Aaand another! Or at least the code for the last app to play around with ðŸ™‚ I couldn’t really see much on the video, though I will have a go now that I know a bit more about how it works.