We’ve tried buying ads on AdMob for all three [...]]]> Part three of my series on various things we learned about selling on the iOS App Store. (Part 1, part 2)

Don’t pay for AdMob clicks!

We’ve tried buying ads on AdMob for all three of our games, and, like just about any other account on the web that we’ve read said, the results were abysmal. It wasn’t for lack of trying to be smart, either:

We restricted the geographic locations to US/Canada/UK/Australia (because why pay for Chinese clicks on an English ad), we restricted the target group to what we thought were our core demographics, and we experimented with different ad texts (we figured if we could get only geeks to click on our “CellShades” ad and scare everyone else off, our conversion rate per click would be so much higher). All to no avail. Our click-through rate was always in the low to mid 0.x%, but our conversion rate after the click-through was so low that for every $50 we spent, the bump in downloads was negligible (as in practically not measurable), even with our free app “CellShades“.

My conclusion is that AdMob may have it’s place if you have a huge marketing budget and you need to generate just enough extra sales to make it from position 26 to 25 in the charts. As you can generate a lot of impressions for little money, it may also be worthwhile to use AdMob to A/B test your app’s name and icon. I’ll share our experience with that in another post. But for an indie dev on a budget, buying ads to sell your game is a complete waste of money.

Setting your app free for a day will result in massive downloads

When you set a paid app to free, a slew of app shopping sites will automatically pick up on the fact and spread the news over the web and social media sites. The effect can be in the tens of thousands of additional downloads over the next couple of days, with download numbers quickly tapering back to normal free app levels over the course of a week or so.

The effect is repeatable, but if the intervals between repetitions aren’t very long (several months to half a year, it seems to me), its intensity will gradually fade away. Repeating the experiment after two weeks, for instance, probably won’t produce a significant amount of downloads. Also keep in mind that some sites will offer a complete history of your app’s pricing, so if you do this frequently, potential customers might pick up on it.

Don’t expect the additional downloads to significantly increase sales after you switch the app back to paid, though. The people who visit automated price drop reporting sites don’t seem to be particularly willing to pay for apps they see there that have reverted back to full price.
Also, as the downloads you get are very untargeted, a lot of developers have reported that the users who download your app when it’s free are statistically more likely to leave negative reviews in the App Store (something we have not personally experienced).

So why do it? In-app purchases are one good reason. Another one would be that if you have the kind of app that people like to show off to their friends, the amount of additional word of mouth might well make it worth it.

Read up on App Store SEO!

Users searching the App Store can contribute a significant portion of revenue over your app’s lifetime. Whether or not your app is among the results for any particular search depends on your title, the keywords you specify in iTunes Connect and the names of any in app purchases. Interestingly, your app’s description text does not contribute to its search engine placement.

Apple only gives you 100 bytes for your app’s keywords and you can only update them with new versions of your binary. Make the most of them! People have experimented with different keywords and shared lessons about SEO on the App Store, so read their accounts (here’s an interesting four-parter: part 1, part 2, part 3, part 4)!

Use commas (instead of commas and spaces) to separate keywords to save room. All other things equal, shorter keywords are better than longer ones. Use small updates to your app to optimize your keywords and periodically check where you rank for any of your search terms. Use services like http://www.appcod.es/appsearch/ to see what keywords your competitors might be using.

It’s also worth pointing out that your keyword ranking depends on factors such as how many downloads you’ve recently had (and your app’s ratings), so if you’ve previously optimized out a popular keyword because your rank was too low, and your app picks up steam, you may want to reconsider.

User behavior tracking is easy!

There are plenty of services on the web that let you add tracking code to your iOS application and give you statistics on how users interact with it. If you’re writing a game, how many people get through the first level after the tutorial? Is there a particular puzzle at which people tend to give up? If you change the upsell screen in your lite version, or the description of your in-app purchase, will more or less users make the purchase?

When we started out, we refrained from tracking user behavior, because it seemed like a lot of work. As it turns out, you can sign up for a free service and add the relevant functionality in an hour or two. We’ve been using the free service of Playtomic ourselves, a cross-platform API geared towards games, which has the nice benefit of also being compatible with C++, Flash and Android applications. There are other services as well, but I haven’t yet tried any of them.

Playtomic is by no means perfect: you have to resort to creatively naming your tracking events in order to do A/B testing (i.e. showing different portions of your users different versions of the same thing – such as an upsell screen –, and measuring the difference in reaction), and I really miss the ability to assign attributes to users and track them separately based on them (for instance, I’d like to track the behavior of pirates, new users, power users and users that pay for an iAP separately). The service has also had some downtime and a few instances of quirky reporting in the three months that we’ve been using it, but I’d chalk that up to it being relatively young.

On the upside, the analytics are updated on a nearly real-time basis, the API is dead simple to add, and you can’t beat the price of free.

In this series of posts, I’ll try to summarize some of the things we learned over the year. This is by no means a comprehensive list – just a series of items that I would have liked to read about when we worked on our first app. Part 2 is here, and part 3 here!

Be wary of outdated information

The App Store is an environment that is ever changing, in terms of the competition, in terms of what users like in an app, and in terms of the inner workings of the store and the rules set forth by Apple.

While the early days of iPhone development yielded story after story of lone indie devs taking the charts by storm and then using the momentum of high ranks to stay there, these days dozens of big fish wield enormous marketing budgets to fight for the coveted top 25 spots, and surprise hits are considerably less common. While players might still find the idea of collecting coins to buy little hats for their avatars fun now, they may just as well be tired of it in another six months. And while giving out promo codes in exchange for user reviews or making your game free for a day to rise up the charts used to work great a couple of years ago, Apple has changed the way the App Store works to remove these ways of gaming the system.

The bottom line is, you should take any information about what works well on the App Store that is older than six months with a large grain of salt; and much more so, if the information could be considered to be exploiting loopholes in the store. Any advice older than a year is probably not worth taking. To save you from scrolling up: this article was written in May 2012.

I’d also suggest considering possible future developments before resorting to black hat techniques such as buying fake user reviews. Apple has explicitly stated that such behavior may be grounds for terminating your account, and just because they aren’t cracking down on the cheaters now doesn’t mean they can’t or won’t.

If you pick a name for your app, you have 180 days to finish it.

This came as quite a surprise for us (guess whether it was a pleasant one!):

From the day you create your app in iTunes Connect, you have 180 days to upload a binary. After 150 days or so, you’ll get a few reminder e-mails telling you that if you don’t finish your app, the name will be available for another developer to use. What they don’t tell you, though, is that you won’t be able to re-register the name yourself (as opposed to an expired domain name, which becomes available to the world, including you).

Since you need to create and name your app in iTunes Connect in order to add things like Game Center support or iAPs, this is something to really look out for if your project is big or you’re doing it on the backburner.

We’ve run into this with our own game “Coign of Vantage”. To be fair, we sent Apple an e-mail stating our case at the time, and we did receive a 120 day extension. But they took more than a month to reply, and just because they resolved our issue doesn’t mean they will do the same for you.

If your project could take more than half a year to complete, it might make sense to start it under a different name, which you can delete from iTunes Connect later on. (Disclaimer: I haven’t checked whether this is against any of Apple’s rules)

Update: Another way that appears to still work is to upload a dummy binary when the deadline arrives and then self-reject it.

Don’t mess up your placement in the “New Releases” list

While its importance has declined a lot in the past couple of years, the “new releases” list in the App Store gives you a few guaranteed sales and some muchly needed initial exposure. Since you’ll need all the momentum you can get, make sure your app doesn’t hit one of the few remaining obstacles that mess up its placement on the list:

If your app is approved and released after the release date you specify, it enters the “new releases” list a few pages back (which means nobody will find it), so when submitting your app for review, pick a release date a few months in the future. Only once the app has been approved should you set the date back to the actual date you want to release it. Set its release to at least one full day after the current date.

I’ve also heard that before your app is released, you can only change the release date once without messing up your “new releases” placement. I have no idea whether that’s true, but I’d rather play it safe. Your app will only be placed in the “new releases” list of the primary category you choose. The secondary category will be ignored.

The “new releases” list is sorted by alphabet by day, so all other things equal, apps that start with letters A to C will have a slight edge over other apps which may appear on the second or third page.

Submit your first update on the day you launch

Whether you’ll spend them on review sites, giveaways, or your twitter followers, during launch week you’ll quickly run out of the 50 promo codes Apple provides for your app. As you get a fresh batch of codes with every update and an update normally takes about a week to pass Apple’s review, you’ll want your first update to go through as quickly as possible. Keep a minor feature or a couple of levels out of the initial release, and submit your version 1.1 the day your app hits the App Store.

If your release date is after the approval date, it might even be worth submitting your first version before the app goes live, although I don’t know if that behavior could raise any red flags at Apple.

Update: You can now also buy promo codes for your own game as gifts (effectively at 30% the price of your app, as you earn 70% back for the sales), and hand them out. These come with the advantage that people who use them can leave a review on the App Store, which you can’t do with regular promo codes.

Expedited Apple review requests

Any update you submit to Apple will go through the same review process as a new app, which is to say, it will take about a week to go through, provided everything checks out okay. If you discover a critical bug on launch day, a week could be what makes or breaks your release buzz. For such situations, Apple gives you the option of requesting an expedited review at this URL: https://developer.apple.com/appstore/contact/appreviewteam/index.html

Expedited reviews are granted on a limited basis, so be prepared to state a good reason why your update should get preferential treatment. Also, I wouldn’t be surprised if crying wolf too often could get your account flagged.

Use a third-party tool to generate sales reports

Let’s face it, as far as monitoring your sales and download data is concerned, iTunes Connect is rubbish. You can either look at graphs for the downloads, sales, iAPs and updates of all your apps lumped together, or you can download an Excel sheet with raw data. You should also be aware that iTunes Connect only keeps track of your sales during the past couple of months. If you want to look at sales stats from last year, you’re out of luck.

There are numerous free services around that provide much better data, AppAnnie being the one we are using. The way this works is you hand AppAnnie your iTunes Connect credentials, and once a day the service pulls the sales data from there, combining it with data they gather from the web.

You can then get a daily report of your exact revenue (from all App Stores combined) and downloads separated by app, your ranking in the various stores, any new reviews of your apps, and other things. You can also check out sales graphs that go back for as long as AppAnnie had access to your iTunes Connect account.

CellShades is an experimental little app, similar to “Cellular Automata“, in which you spill liquid which provides energy for cells to spawn and harvest. These cells follow a handful of very simple rules (such as that they will always [...]]]>
Last week, we released a new iPhone/iPad app named CellShades.

CellShades is an experimental little app, similar to “Cellular Automata“, in which you spill liquid which provides energy for cells to spawn and harvest. These cells follow a handful of very simple rules (such as that they will always move towards adjacent positions which they perceive as more energetic, or that they will perish and dissolve into liquid if they can’t keep up their energy levels) to produce a wide variety of different behaviors. The examples that ship with the app range from crashing waves to flowering structures to larger microorganisms crawling across the screen, giving a taste of the huge spectrum of dynamics possible with the “simulation”.

The app comes with 9 free preset behaviors. A one Dollar (0.79€, etc.) in-App purchase unlocks the ability to manipulate the simulation’s underlying parameters and save your own behaviors.

Technology and libraries

We wrote CellShades in Objective-C, with the main simulation parts being mostly C. I started out writing the app in cocos2d, mainly because it provided a very quick way of setting up everything so that the app could draw and update its simulation and respond to user input. The main GUI of the app consists of UIKit elements residing on top of the cocos2d surface.

As a relative beginner to Objective-C, I was a little wary of the cocos2d/UIKit mix and what pitfalls it might harbor, but I have to say that everything went very smoothly and I wouldn’t hesitate to go that route again (a good starting tutorial is here).

We have some sharing features in CellShades which were implemented using the ShareKit library and which enable users to put screenshots into their photo album or share them via Facebook, Twitter or Tumblr. That took a little longer than expected because the library wouldn’t quite work out of the box – specifically, I recall hunting down a couple of things to change for Twitter compatibility on the forums. We also didn’t like ShareKit’s default way of presenting an ActionSheet, complete with a ShareKit settings button, so a little more work was spent replacing that with our own GUI which would then create and share items manually. But all in all, these problems were pretty minor and the end result is a very re-useable bit of code, so no complaints there.

We also used Playtomic‘s analytics API for tracking, gathering information such as how often the app is launched before the in-app purchase is made or how often the info-screens are opened. As this is the first time we’re using it and the app has just been released, the jury on that is still out, and I’ll reserve a detailed account of our experience with Playtomic for a future post.

Finally, if you’d like to see the app in action, click here for the App Store link!

Basically, what we’re going to do [...]]]> In the first two tutorials of this series on dynamic audio in AS3, we’ve covered pretty much everything that Flash’s realtime sound API offers us. Let’s put all of it to use and benefit humankind by building a little app that will turn your voice into a horrible robot!

Pictured: Code sample 5. (image source: http://www.dieselsweeties.com)

Basically, what we’re going to do is take input from the microphone in a stream of sound samples, try to come up with something interesting to do with the samples, and then send the samples on to the sound card’s output.

Rather than building the app so that the user gets record and play buttons with which they can record a take and play back the processed version, we’ll build a real-time effect and process and play back the sound as it comes in. There are two reasons I want to go this route:

First, there’s less chrome involved that doesn’t add to the subject (GUI, application state and such).

Second, real-time processing is actually harder and more interesting than processing pre-recorded audio, because it poses the question how we can make (reasonably) sure that we have received enough input from the mic whenever we’re asked to fill a new output buffer. I’ll go over the intricacies of that in the next two sections – if you want to get right to the part where we do nasty things to your vowels, feel free to skip these and come back to them later.

Buffer syncing woes, in theory…

We have a microphone that periodically provides input to a buffer, and sound output which periodically asks us to fill a buffer with new samples. Let’s take a look at why syncing the two isn’t that trivial:

Suppose that, for argument’s sake, your output buffer is 3500 samples large, and the microphone dispatches a new SampleDataEvent whenever it has 3000 new samples available for you. If you start both the output sound instance and the microphone input at the same time, what’s going to happen?

At t=0 (the time unit here is single samples), you’re asked to create a 3500 sample output buffer, but you have collected no input yet! So, let’s say you fill the output buffer with zeros.

At t=3000, 3000 samples come in from the microphone, so you process and store these to use later. At t=3500, the sound card needs 3500 new samples, but you have only collected 3000 from the input so far! Do you wait until the next output cycle (introducing a further 3500 sample lag!), or do you send 500 zero-samples followed by 3000 processed input samples to the output buffer and continue from there?

Let’s say you do the latter: At t=3500, your input buffer is now of size 0. At t=6000, 3000 new samples come in from the input, so you add them to your input buffer (which is now 3K). However, at t=7000, the output buffer is empty again, and you’ll need 3500 input samples to fill it, but you only have 3000 available. If you pad the rest with zeros, you’ve introduced stuttering. What’s worse, the math actually works out so that the stuttering will reoccur periodically (if you go the route of adding to the output whatever number of samples you have at the time and padding the rest with zeros): at t=7000, your input is at 0 again. T=9000, 3K samples come from the input, but at t=10500, 3.5K samples are needed again for the output and you only have 3K…

The above is what happens when we start both mic input and sound output at the same time, and the output buffer is larger than the input buffer. Let’s look at a case where both are exactly the same size, say 3K:

At t=0, the output buffer needs 3K samples, but you don’t have any yet. At t=3K, 3K samples come in from the mic input, and the output buffer needs 3K new samples. So you process the input, send it to the output, and your input storage remains at 0. At t=6K, the same thing happens again, and 3K samples come in just before the output buffer needs 3K new output samples, again setting the input buffer to size 0.

The “just before” in the last sentence means trouble! What happens, if, by whatever operating system hiccup or timer inconsistency or hardware inconsistency, the SampleDataEvent that asks for new output suddenly gets called before the SampleDataEvent that provides new input? You get a 3000 sample pause in your output!

Finally, let’s look at a case where the output buffer is smaller than the input buffer size. Let’s say the microphone data comes in every 3500 samples, and your output buffer is 3000 samples in size (and you’re starting both at the same time).

At t=0, you’re expected to send 3000 samples to the output, but you haven’t received any from the mic, so you send 0s. At t=3K, the same thing happens again. At t=3.5K, 3.5K samples come in from the mic, so you process and store them. At t=6K, you need to send 3000 samples to the output, so you take them from your input storage, leaving 500 more samples in store. At t=7K, the mic sends you a further 3.5K samples, so your input buffer is now 4K samples big. At t=9K, the output requests 3K, leaving 1K in your input buffer.

I’ll cut this a little short and give you a table of what happens to the input buffer’s size at subsequent points in time.

t=10.5K → 4.5K (+3.5K came from input)
t=12K → 1.5K (-3K went to output)
t=14K → 5K (in)
t=15K → 2K (out)
t=17.5K → 5.5K (in)
t=18K → 2.5K (out)
The interesting thing actually happens at t=21K. In theory, at this point, you should both receive 3.5K of input samples and supply 3K of output samples! The problem is that, if you’re asked to produce the output before the input arrives (and there’s really no guarantee which will happen first), you are 500 samples short and you’ll have a gap in your playback!

… and in practice!

Clearly, if we’re transferring mic input to audio output, we need to accumulate some input and buffer it before we start the output. How much? To be frank, I really don’t know.

For starters, how much sample data will we get from the microphone at once? The answer is, that’s actually inconsistent! On my Mac, I’m getting new mic data every 1024 samples, but just occasionally the mic skips a SampleDataEvent entirely, and the next one comes with 2048. I’ve even seen hiccups where I get 4096 samples at once, although to be fair, I’ve only seen these near the application’s initialization.

Also, I have found no information on whether the 1024 sample interval is a target for all platforms, or whether the input intervals are different depending on your operating system, or even your sound card. If that information isn’t explicitly stated anywhere, it’s probably a good idea to treat it as undefined, and therefore subject to change.

So how do we proceed? I can think of two strategies:

a.) Buffer up an empirically determined amount of input samples before even starting the output, erring on the side of caution. The amount you want to pre-buffer is at least the size of one complete output buffer plus one complete input. Suppose your output buffer is 2048 samples and new data comes in every 1024 samples – you’d buffer up 3072 samples before starting playback. Since we already know that occasionally new mic input only comes in every 2048 samples, let’s be safe and make the buffer at least 4096. To be sure, you’re adding 4096/44100 = 92ms of latency between when a sample comes in from the mic’s SampleDataEvent and when it goes into the output sample data (and that’s on top of all the other latencies, including another guest to the party, which is your sound card’s input lag), but at least you’re fairly safe as far as buffer underflows are concerned.

b.) Start with a small safety buffer and monitor for underflows (i.e. instances where you have no input data left so you need to pad with zeros). Whenever an underflow occurs, increase the safety buffer’s maximum size and buffer until it is full again before continuing playback. This will be sure to have stutters in the first few seconds or so, but it should converge on the minimum “safe” latency (where “safe” means safe until something that holds up the input happens that didn’t happen before).

Those are the two avenues I’ve come up with. If you have a different suggestion, please leave a comment!

Personally, I’d argue that route a.) is almost always preferable. It works, it’s easier to implement, and if your objective is low latency input processing in Flash, then I’m afraid you’ve already lost. We might as well make the application perform with as few gaps as we can.

Enough already, let’s make some noise!

The following code sample sets up input and output SampleDataEvent handlers and pre-buffers at least the amount of samples specified in the MIN_SAFETY_BUFFER constant (more if MIN_SAFETY_BUFFER is not an integer multiple of the microphone’s input buffer size). In order to make this a complete application, it would be a good idea to handle microphone status events as well, so as to deal with users that don’t allow you to gather microphone input, but we’ll leave that out for now, to keep the example short and simple.

You can actually put the code on the timeline in Flash! Make sure you put on your headphones so you don’t get feedback.

import flash.media.Microphone;
import flash.events.SampleDataEvent;
import flash.media.Sound;

/*
Example 1:
Microphone input to audio output
*/

const BUFFER_SIZE:int = 2048; // output buffer size
const MIN_SAFETY_BUFFER:int = 1024; // minimum collected input before output starts

var outputActive:Boolean = false; // will be set to true once MIN_SAFETY_BUFFER samples have been collected.

var mic:Microphone = Microphone.getMicrophone();
mic.rate = 44;
mic.setSilenceLevel(0); // you need to set this, or else the mic will stop sending data when it detects prolonged silence!
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);

var inputBuffer:Vector.<Number> = new Vector.<Number>(); // buffer in which we'll store input data

var playbackSound:Sound = new Sound();
playbackSound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
playbackSound.play();


function micSampleDataHandler(event:SampleDataEvent):void 
{
  while(event.data.bytesAvailable)
  {
    var sample:Number = event.data.readFloat(); // microphone input is mono! 
    inputBuffer.push(sample);
  }
  if (!outputActive && inputBuffer.length >= MIN_SAFETY_BUFFER)
  {
    trace("starting playback!");
    outputActive = true;
  }
}

function soundSampleDataHandler(event:SampleDataEvent):void
{
  var outputBuffer:Vector.<Number>;
  // move samples from input to output buffer:
  if (outputActive)
  {
    // if playback is enabled, take BUFFER_SIZE number of samples from the input buffer...
    outputBuffer = inputBuffer.splice(0, BUFFER_SIZE);
    if (outputBuffer.length < BUFFER_SIZE)
    {
      trace("buffer underflow!");
      while (outputBuffer.length < BUFFER_SIZE) outputBuffer.push(0);
    }
  } else
  {
    // ... otherwise create an empty output buffer of the right size
    outputBuffer = new Vector.<Number>(BUFFER_SIZE);
  }
  
  // process samples and add them to the SampleDataEvent's data.
  for (var i:int=0; i<BUFFER_SIZE; i++)
  {
    var currentSample:Number = outputBuffer[i];
    // do something interesting with the sample here!

    event.data.writeFloat(currentSample); // left channel
    event.data.writeFloat(currentSample); // right channel
  }
}

									

Right at the bottom of the code, there’s a line where the currentSample variable is set, just before it is added to the ByteArray that is sent to the sound card: var currentSample:Number = outputBuffer[i];

Here would be the best place to do something interesting to each sample.

What should we do? It’s up to you to experiment! Here are a few ideas to get you started:

1.) Add a very short delay (a few milliseconds) with feedback! The result is basically a comb filter with a very metallic sound. One way to implement this is to have another Vector.<Number>, and use it as a queue. For each sample, if the queue has reached the length necessary to begin the delay effect, take the first element off the queue, multiply it with the feedback factor and add it to the current sample. Finish processing the current sample, then add it to the end of the queue.

import flash.media.Microphone;
import flash.events.SampleDataEvent;
import flash.media.Sound;

/*
Example 2:
Delay / comb filter
*/

const BUFFER_SIZE:int = 2048; // output buffer size
const MIN_SAFETY_BUFFER:int = 1024; // minimum collected input before output starts

// buffer used for comb filter effect
var delayQueue:Vector.<Number> = new Vector.<Number>();
var delayLength:int = 500; // length of the delay in samples
var delayFeedback:Number = 0.9; // strength of the delay effect.

var outputActive:Boolean = false; // will be set to true once MIN_SAFETY_BUFFER samples have been collected.

var mic:Microphone = Microphone.getMicrophone();
mic.rate = 44;
mic.setSilenceLevel(0); // you need to set this, or else the mic will stop sending data when it detects prolonged silence!
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);

var inputBuffer:Vector.<Number> = new Vector.<Number>(); // buffer in which we'll store input data

var playbackSound:Sound = new Sound();
playbackSound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
playbackSound.play();


function micSampleDataHandler(event:SampleDataEvent):void 
{
  while(event.data.bytesAvailable)
  {
    var sample:Number = event.data.readFloat(); // microphone input is mono! 
    inputBuffer.push(sample);
  }
  if (!outputActive && inputBuffer.length >= MIN_SAFETY_BUFFER)
  {
    trace("starting playback!");
    outputActive = true;
  }
}

function soundSampleDataHandler(event:SampleDataEvent):void
{
  var outputBuffer:Vector.<Number>;
  // move samples from input to output buffer:
  if (outputActive)
  {
    // if playback is enabled, take BUFFER_SIZE number of samples from the input buffer...
    outputBuffer = inputBuffer.splice(0, BUFFER_SIZE);
    if (outputBuffer.length < BUFFER_SIZE)
    {
      trace("buffer underflow!");
      while (outputBuffer.length < BUFFER_SIZE) outputBuffer.push(0);
    }
  } else
  {
    // ... otherwise create an empty output buffer of the right size
    outputBuffer = new Vector.<Number>(BUFFER_SIZE);
  }
  
  // process samples and add them to the SampleDataEvent's data.
  for (var i:int=0; i<BUFFER_SIZE; i++)
  {
    var currentSample:Number = outputBuffer[i];
    // delay effect / comb filter:

    // if the delay queue has reached its target length, take the sample at the beginning and 
    // mix it with currentSample
    if (delayQueue.length > delayLength) 
    {
      var delayedSample:Number = delayQueue.shift();
      currentSample += delayedSample*delayFeedback;
    }

    // push current sample to the delay queue's end
    // note: we're adding the already processed sample back into the queue, so we get
    // feedback of feedback, etc.
    delayQueue.push(currentSample);

    event.data.writeFloat(currentSample); // left channel
    event.data.writeFloat(currentSample); // right channel
  }
}

									

2.) Perform frequency shifting! Frequency shifting is actually really easy to do – just multiply the samples with a sine wave. (The rather complicated technical explanation for why this works is that multiplication in the time domain is the same as convolution in the frequency domain, and in the frequency domain, a sine wave is just an ideal impulse, offset by its frequency – hopefully I’ll be able to make this clearer in a future article!)
Frequency shifting adds non-harmonic content and gives a pretty cool metallic character to human speech. Note that it’s different from pitch shifting which basically multiplies all the frequencies that make up a given sound with a constant, whereas frequency shifting adds a constant offset to all the frequencies in a sound.

import flash.media.Microphone;
import flash.events.SampleDataEvent;
import flash.media.Sound;

/*
Example 3:
Frequency shifter
*/

const BUFFER_SIZE:int = 2048; // output buffer size
const MIN_SAFETY_BUFFER:int = 1024; // minimum collected input before output starts

var freqShiftPhase:Number = 0; // phase of the sine wave used for frequency shifting
var freqShiftDeltaPhase:Number = 75*2*Math.PI/44100; // phase increase per sample

var outputActive:Boolean = false; // will be set to true once MIN_SAFETY_BUFFER samples have been collected.

var mic:Microphone = Microphone.getMicrophone();
mic.rate = 44;
mic.setSilenceLevel(0); // you need to set this, or else the mic will stop sending data when it detects prolonged silence!
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);

var inputBuffer:Vector.<Number> = new Vector.<Number>(); // buffer in which we'll store input data

var playbackSound:Sound = new Sound();
playbackSound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
playbackSound.play();


function micSampleDataHandler(event:SampleDataEvent):void 
{
  while(event.data.bytesAvailable)
  {
    var sample:Number = event.data.readFloat(); // microphone input is mono! 
    inputBuffer.push(sample);
  }
  if (!outputActive && inputBuffer.length >= MIN_SAFETY_BUFFER)
  {
    trace("starting playback!");
    outputActive = true;
  }
}

function soundSampleDataHandler(event:SampleDataEvent):void
{
  var outputBuffer:Vector.<Number>;
  // move samples from input to output buffer:
  if (outputActive)
  {
    // if playback is enabled, take BUFFER_SIZE number of samples from the input buffer...
    outputBuffer = inputBuffer.splice(0, BUFFER_SIZE);
    if (outputBuffer.length < BUFFER_SIZE)
    {
      trace("buffer underflow!");
      while (outputBuffer.length < BUFFER_SIZE) outputBuffer.push(0);
    }
  } else
  {
    // ... otherwise create an empty output buffer of the right size
    outputBuffer = new Vector.<Number>(BUFFER_SIZE);
  }
  
  // process samples and add them to the SampleDataEvent's data.
  for (var i:int=0; i<BUFFER_SIZE; i++)
  {
    var currentSample:Number = outputBuffer[i];
    
    freqShiftPhase += freqShiftDeltaPhase;
    currentSample *= Math.sin(freqShiftPhase);
    
    event.data.writeFloat(currentSample); // left channel
    event.data.writeFloat(currentSample); // right channel
  }
}

									

3.) Take the first two ideas and add low frequency oscillators to them. If you modify the delay time of the comb filter (code example 2) with an LFO, you get your basic flanger! If you modify the frequency of the sine wave you multiply the input with (code example 3), you get a bouncy, cartoony effect, of which I’m not quite sure whether it actually has a name.
The following code produces a flanger effect by moving back and forth the index of the sample in the delay queue, the one that we add in as feedback. Note the somewhat dirty, distorted sound of the effect: That’s due to the fact that we perform no interpolation whatsoever. In the future, I’ll write a separate tutorial on flanging and chorus effects in which we’ll clean this up, but this is Robot Building 101, so a little dirt won’t hurt us!

import flash.media.Microphone;
import flash.events.SampleDataEvent;
import flash.media.Sound;

/*
Example 4:
Adding an LFO to produce flanging
*/

const BUFFER_SIZE:int = 2048; // output buffer size
const MIN_SAFETY_BUFFER:int = 1024; // minimum collected input before output starts

// buffer used for comb filter effect
var delayQueue:Vector.<Number> = new Vector.<Number>();
var delayLength:int = 100; // length of the delay in samples
var delayFeedback:Number = 0.8; // strength of the delay effect.
// lfo to transform the comb filter into a flanger
var lfoPhase:Number = 0; // phase of the LFO
var lfoDeltaPhase:Number = 0.5*2*Math.PI/44100; // phase increase per sample
var lfoModStrength:Number = 20; // strength of the lfo, given in +/- sample offset

var outputActive:Boolean = false; // will be set to true once MIN_SAFETY_BUFFER samples have been collected.

var mic:Microphone = Microphone.getMicrophone();
mic.rate = 44;
mic.setSilenceLevel(0); // you need to set this, or else the mic will stop sending data when it detects prolonged silence!
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);

var inputBuffer:Vector.<Number> = new Vector.<Number>(); // buffer in which we'll store input data

var playbackSound:Sound = new Sound();
playbackSound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
playbackSound.play();


function micSampleDataHandler(event:SampleDataEvent):void 
{
  while(event.data.bytesAvailable)
  {
    var sample:Number = event.data.readFloat(); // microphone input is mono! 
    inputBuffer.push(sample);
  }
  if (!outputActive && inputBuffer.length >= MIN_SAFETY_BUFFER)
  {
    trace("starting playback!");
    outputActive = true;
  }
}

function soundSampleDataHandler(event:SampleDataEvent):void
{
  var outputBuffer:Vector.<Number>;
  // move samples from input to output buffer:
  if (outputActive)
  {
    // if playback is enabled, take BUFFER_SIZE number of samples from the input buffer...
    outputBuffer = inputBuffer.splice(0, BUFFER_SIZE);
    if (outputBuffer.length < BUFFER_SIZE)
    {
      trace("buffer underflow!");
      while (outputBuffer.length < BUFFER_SIZE) outputBuffer.push(0);
    }
  } else
  {
    // ... otherwise create an empty output buffer of the right size
    outputBuffer = new Vector.<Number>(BUFFER_SIZE);
  }
  
  // process samples and add them to the SampleDataEvent's data.
  for (var i:int=0; i<BUFFER_SIZE; i++)
  {
    var currentSample:Number = outputBuffer[i];
    // flanger

    // if the delay queue has reached its target length, take the sample at the beginning and 
    // mix it with currentSample
    if (delayQueue.length > delayLength) 
    {
      delayQueue.shift(); // remove first sample
      
      lfoPhase += lfoDeltaPhase;
      var delayedSample:Number = delayQueue[ Math.floor( Math.sin(lfoPhase)*lfoModStrength )+lfoModStrength ];
      
      currentSample += delayedSample*delayFeedback;
    }

    // push current sample to the delay queue's end
    delayQueue.push(currentSample);

    event.data.writeFloat(currentSample); // left channel
    event.data.writeFloat(currentSample); // right channel
  }
}

									

Now put all of these together! Note what happens when you change the order of operations (it should have an effect, because the frequency shifting is a non-linear operation)! Experiment and find other ways to put dings and dents into the input material! What happens to the sound when you apply a non-linear function to each sample, such as taking its cube (hint: distortion!)? What happens to the sound when you change your frequency shifting implementation to use a triangle wave instead of a sine (hint: I have no idea, go try it out!)?

I’ll leave you with another combination of the above ideas, just as another starting point. The final code sample combines a flanger with a frequency shifting effect. Let’s hope it never achieves sentience!

import flash.media.Microphone;
import flash.events.SampleDataEvent;
import flash.media.Sound;

/*
Example 5:
Hacking away...
*/

const BUFFER_SIZE:int = 2048; // output buffer size
const MIN_SAFETY_BUFFER:int = 1024; // minimum collected input before output starts

// buffer used for comb filter effect
var delayQueue:Vector.<Number> = new Vector.<Number>();
var delayLength:int = 800; // length of the delay in samples
var delayFeedback:Number = 0.8; // strength of the delay effect.
// lfo to transform the comb filter into a flanger
var lfoPhase:Number = 0; // phase of the LFO
var lfoDeltaPhase:Number = 20*2*Math.PI/44100; // phase increase per sample
var lfoModStrength:Number = 100; // strength of the lfo, given in +/- sample offset

var freqShiftPhase:Number = 0; // phase of the sine wave used for frequency shifting
var freqShiftDeltaPhase:Number = 1800*2*Math.PI/44100; // phase increase per sample


var outputActive:Boolean = false; // will be set to true once MIN_SAFETY_BUFFER samples have been collected.

var mic:Microphone = Microphone.getMicrophone();
mic.rate = 44;
mic.setSilenceLevel(0); // you need to set this, or else the mic will stop sending data when it detects prolonged silence!
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);

var inputBuffer:Vector.<Number> = new Vector.<Number>(); // buffer in which we'll store input data

var playbackSound:Sound = new Sound();
playbackSound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
playbackSound.play();


function micSampleDataHandler(event:SampleDataEvent):void 
{
  while(event.data.bytesAvailable)
  {
    var sample:Number = event.data.readFloat(); // microphone input is mono! 
    inputBuffer.push(sample);
  }
  if (!outputActive && inputBuffer.length >= MIN_SAFETY_BUFFER)
  {
    trace("starting playback!");
    outputActive = true;
  }
}

function soundSampleDataHandler(event:SampleDataEvent):void
{
  var outputBuffer:Vector.<Number>;
  // move samples from input to output buffer:
  if (outputActive)
  {
    // if playback is enabled, take BUFFER_SIZE number of samples from the input buffer...
    outputBuffer = inputBuffer.splice(0, BUFFER_SIZE);
    if (outputBuffer.length < BUFFER_SIZE)
    {
      trace("buffer underflow!");
      while (outputBuffer.length < BUFFER_SIZE) outputBuffer.push(0);
    }
  } else
  {
    // ... otherwise create an empty output buffer of the right size
    outputBuffer = new Vector.<Number>(BUFFER_SIZE);
  }
  
  // process samples and add them to the SampleDataEvent's data.
  for (var i:int=0; i<BUFFER_SIZE; i++)
  {
    var currentSample:Number = outputBuffer[i];

    
    if (delayQueue.length > delayLength) 
    {
      delayQueue.shift(); // remove first sample
      
      lfoPhase += lfoDeltaPhase;
      var delayedSample:Number = delayQueue[ Math.floor( Math.sin(lfoPhase)*lfoModStrength )+lfoModStrength ];
      
      currentSample += delayedSample*delayFeedback;
      delayQueue.push(currentSample);

      freqShiftPhase += freqShiftDeltaPhase * Math.sin(lfoPhase*0.4);
      
      currentSample = currentSample*0.6 + 0.4*Math.sin(freqShiftPhase)*currentSample;
      
    } else delayQueue.push(currentSample);


    event.data.writeFloat(currentSample); // left channel
    event.data.writeFloat(currentSample); // right channel
  }
}

									

As [...]]]> In this second installment of my series of tutorials on dynamic sound in ActionScript, I’ll discuss the different parts of the sound API and show you how to extract single samples from a sound that’s in memory or coming from the microphone, as well as how to generate simple dynamic audio in real time.

As I wrote in part 1, the sound API introduced in Flash Player 10 is essentially just a set of methods and classes that let you access individual samples in a sound. Just as the introduction of the BitmapData class enabled you to manipulate and read pixels in bitmap images and from a webcam’s input, the sound API lets you process sound in memory or coming from microphone input, at the sample level.

There are two basic ingredients you need to understand in order to cook up dynamic audio – ByteArrays and SampleDataEvents:

ByteArrays

ByteArrays are the format in which you’ll be dealing with the raw data. Whenever you read the samples contained in an audio clip, write samples to the sound output or receive new samples from the mic, you’ll be working with a ByteArray instance.

If you’ve never had to deal with ByteArrays in AS3 before, don’t fret – in practice you’ll usually just read and write Numbers from and to them, or convert them to Vector.<Number>s to do any advanced work.

A ByteArray is basically an array of values of larger types (such as Numbers or even serialized Objects), sliced up into bytes and packed together into one big stream. ByteArrays store an index into the data (called the position), which is automatically incremented whenever you read from or write into the stream. So, for instance, if you have a ByteArray and you call its readFloat() method, you’ll get back a 32 bit (= 4 byte) Number value, giving a float representation of whatever the next 4 bytes in the stream contain, and the position is then incremented by 4. The next time you call readFloat(), the next floating point Number value will be returned.

A ByteArray in Flash is essentially a stream in which arbitrary data can be stored together. Use methods like readFloat(), readDouble(), etc. to extract contents at the current position.

ByteArrays allow you to work very close to the metal (by AS3′s standards), and they provide a bit of additional useful functionality, such as data compression. For our purposes, though, all you should need to know is that you can write the next sample into a ByteArray by using writeFloat(), and while the position of a ByteArray is smaller than its length (which is also given in bytes), you can always call readFloat() to extract the next sample.

Let’s look at a quick example to show you how this works in practice:

Suppose you have an instance of class Sound, called sound, and suppose you also have an instance of class ByteArray, called data (which you can simply create with data = new ByteArray(); ):

sound.extract(data, sound.length*44100); // writes the sound’s samples into the ByteArray

The first parameter of Sound.extract() is the ByteArray into which you extract the sound’s sample data. If the ByteArray already has data, the extract() method starts overwriting this data at the ByteArray’s current position and appends where needed. The second parameter of Sound.extract() specifies the number of samples to extract. A sample is two 32-bit floating point Numbers, one for each stereo channel. With the exception of mic input, when you’re dealing with dynamic audio in Flash, you’re always dealing with 44.1 kHz stereo audio, given in normalized (i.e. full amplitude == 1.0) 32-bit floating point. Therefore, if sound.length gives you the length of a sound in seconds, sound.length*44100 gives you the number of samples there are in the complete Sound instance.

So, suppose you’ve extracted a sound’s sample data in this manner. How do you access it?

data.position = 0; // reset the ByteArray's index pointer
while (data.bytesAvailable) // while there's more data at the ByteArray's current position:
{
  var leftSample:Number = data.readFloat();
  var rightSample:Number = data.readFloat();
  trace("Sample Data: "+leftSample+" (left), "+rightSample+" (right).");
}
									

This will continue reading from the ByteArray as long as more data is available. Each iteration of the loop, two 32-bit values are read from the ByteArray, one for each stereo channel. In this case, we simply trace them out (note: as this has 44.1K trace calls per second of audio, it will probably cause trouble for sound files that aren’t super-short!), and again note how they stay between -1 and 1, as 1 is the maximum amplitude of a sample!

If we were to do a lot of processing on the sound samples, especially if the operations we performed took multiple input samples into account for any given output sample, we would probably push these into two Vector.<Number> instances, like so:

data.position = 0; // reset the ByteArray's index
var leftChannel:Vector.<Number> = new Vector.<Number>();
var rightChannel:Vector.<Number> = new Vector.<Number>();
while (data.bytesAvailable) // while there's still data in the ByteArray:
{
  leftChannel.push(data.readFloat());
  rightChannel.push(data.readFloat());
}
									

SampleDataEvent

A pre-loaded Sound, such as one instantiated from the library, is already completely in memory, which is why we can extract its complete data at once. With sound that’s coming from the microphone, new samples come in all the time, so we’ll need to process them as they arrive. This (and the creation of dynamic audio output) is where the SampleDataEvent class comes in.

Suppose you want to capture incoming sound data from the microphone, and append it to the leftChannel and rightChannel vectors we instantiated in the last section. Note that these are Vector.<Number> objects, not ByteArrays:

var mic:Microphone = Microphone.getMicrophone();

mic.rate = 44; // sets the mic's sampling rate to 44.1 kHz (note: not 44.0!)
// [...] set various attributes for the microphone, such as echo suppression, etc.
mic.addEventListener(SampleDataEvent.SAMPLE_DATA, micSampleDataHandler);

function micSampleDataHandler(event:SampleDataEvent):void
{
  // event.data is a ByteArray containing sample data!
  while(event.data.bytesAvailable)
  {
    var sample:Number = event.data.readFloat();
    // microphone input is mono!
    leftChannel.push(sample);
    rightChannel.push(sample);
  }
}
									

What’s happening here is that we have an instance of type Microphone dispatching SampleDataEvents, whenever new data comes in from the mic input. This new data comes in the shape of a ByteArray, again with 44.1 kHz 32-bit data (you can actually specify one of a few different possible sampling rates for the mic, however if you want to use the microphone input in dynamic playback, it’s prudent to use the same rate as the sound output will have). Whenever a SampleDataEvent happens, we read in the data as long as there’s more, and append it to the two Vector.<Number> instances we created somewhere outside. Note that input always comes in mono format, so we append the same sample to both channels.

Sound output!

If we’re analyzing a Sound in memory, we simply extract its sample data into a ByteArray. If we analyze sound coming from a microphone, we listen to the SampleDataEvent. With each new SampleDataEvent, new sample data from the microphone is coming in, and we can take this data and append it to a Vector, which will then contain a recording of all microphone input since its instantiation.

On the opposite end, dynamic sound output uses SampleDataEvents as well! The way this works is that you instantiate an empty Sound object, add a SAMPLE_DATA event listener and then call play() on the instantiated sound. The Sound will dispatch the SAMPLE_DATA event whenever it needs more samples to continue playing, at which point your event handler should supply anywhere between 2048 and 8192 new samples (remember that, strictly speaking, a single sample means two floating point Numbers, one for each stereo channel!). If you supply less than 2048 samples, the sound assumes that it is supposed to finish. It plays the last samples you added and then stops.

The number of samples your SAMPLE_DATA event listener generates every time it is called is called the buffer size. A small buffer minimizes the latency, at the cost of increased risk of buffer underflows (the audio stuttering that happens if the application can’t generate enough samples to keep up). As I pointed out in the last article, your latency is unavoidably going to be horrible on some platforms anyway, so in most cases, there’s not much point in sticking to the lowest possible buffer size at all cost. Note that the buffer size doesn’t need to be a power of two but can be any number in range 2048…8192.

With all that out of the way, let’s take a look at some sample code! The following piece of code simply generates stereo noise (at full volume – turn down your headphones!):

var sound:Sound = new Sound();
sound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
sound.play();
function soundSampleDataHandler(event:SampleDataEvent):void
{
  // We can add an arbitrary amount of new sample data to event.data, anywhere between 2048 and 8192.
  // Anything less than 2048, though, and the sound will play until the end and then stop!
  for (var i:int = 0; i<2048; i++)
  {
    event.data.writeFloat(Math.random()*2-1); // random float -1...+1 for left channel
    event.data.writeFloat(Math.random()*2-1); // random float -1...+1 for right channel
  }
}
									

After all we’ve discussed, this should be fairly straight-forward: Whenever the sound buffer is empty, soundSampleDataHandler() is called. This event handler then proceeds to create 2048 samples, each of which contains two random numbers (one per channel), between -1 and +1. These are written to the ByteArray in the SampleDataEvent handled by soundSampleDataHandler().

Suppose we didn’t want to produce random noise; instead, let’s look at what we’d need to do in order to produce a single A1 sine tone at the standard pitch of 440 Hz:

var sound:Sound = new Sound();
sound.addEventListener(SampleDataEvent.SAMPLE_DATA, soundSampleDataHandler);
sound.play();

var currentPhase:Number = 0;
var deltaPhase:Number = 440/44100;

function soundSampleDataHandler(event:SampleDataEvent):void
{
  for (var i:int = 0; i<2048; i++)
  {
    currentPhase += deltaPhase;
    
    // note: this is unoptimized – normally you'd multiply deltaPhase by Math.PI*2 and remove that part here!
    var currentValue:Number = Math.sin(currentPhase*Math.PI*2); 

    event.data.writeFloat(currentValue);
    event.data.writeFloat(currentValue);
  }
}
									

A sine wave has a period of 2*PI, and a 440 Hz tone means that each second the resulting wave needs to repeat 440 times. 440*2*PI divided by the sample rate (=44100) gives us how much we need to increase the phase of the signal per sample.

This is pretty much the basis for the common oscillators you’ll encounter in a software synth, except that it’s usually simpler to have your phase go from 0 to 1 (instead of to 2PI): For a sine wave, you’ll take the sine of the phase * 2PI. For a square wave, you’d simply check if the phase%1 is above or below 0.5 and set 1 or 0 accordingly. From there on out, you should have no trouble creating triangle or sawtooth waves!

In this tutorial, we’ve discussed all major parts of the AS3 dynamic sound API. In the next installment we’ll put it all together and look at how to manipulate microphone input and turn your voice into a horrible robot! Sounds like fun? Stay tuned!