The BlowChucks Wind Instrument Controller

My latest musical instrument contraption

 

Introduction

Over the past couple of years I've been experimenting with electronic wind instruments that mimic the way that a trombone player interacts with his/her instrument. The two instruments I made, the Gordophone and the 3d Trombone, were interesting and fun to build, but there are a couple of problems I haven't been able to solve, and until I solve those problems, the instruments aren't as playable as I want them to be. Because of that, I've gotten kind of stuck and didn't do much music hacking for a while.

To get unstuck, I decided to write a series of posts describing things I've learned while I was building those controllers, which I am still working on. In that series, I've tried to be as hands-on as possible, publishing diagrams that describe the physical construction, wiring, and source code.

Along the way, I wondered if there was a way to make a wind controller that was more accessible and easier to play than the instruments I'd been designing.

Aha! 

 

Coincidentally, my wife and I finally broke down and bought a Wii system for our kids, which came with the standard WiiMote and Nunchuck accessory. The first time I used the Nunchuck, I thought "wow, this would be a pretty cool controller to use for an electronic instrument." After I did a little web research, I discovered that it has:

• 2 momentary pushbuttons
• a joystick
• a 3-axis accelerometer

On top of that, I discovered that it uses a standard i2c interface, that libraries existed to make it work with Arduino, and that you can even use one without hacking off the connector. Many thanks to Tim Hirzel and Tod E. Kurt, for the WiiChuck library, and another thanks to Tod for the WiiChuck Adapter).

I'm a big fan of Onyx Ashanti's work, and I've always been impressed with how his beatjazz controller frees up his hands to move around independently. This lead me to think about using two Nunchucks in a single instrument, in addition to the pressure sensor.

Breadboard Layout

 

The instrument is really simple - only 4 parts are needed. A Freescale pressure sensor (the one I discussed in a previous post), the two nunchucks, and a Teensy 3.0 microcontroller. If you've read my previous posts, you'll note that I'm now using the Teensy 3.0 instead of the Teensy 2.0. I'll discuss why I made that change in the section "Challenges" later in this post.

The wiring is pretty simple:

The pressure sensor gets a reference voltage on pin 2, ground on pin 3, and the output connects to pin 4. The output voltage varies between 0 and the reference voltage depending on the breath pressure (so this part should work fine on either 3.3 or 5 volt controllers, and that's been my experience).

The pressure sensor output goes to the Teensy analog pin 0, which is read to determine how hard the performer is blowing.

Each Nunchuck gets a connection to ground and power, and a distinct i2c bus connection to the Teensy.

Challenges

 

I began work on this project by using a Teensy 2.0 controller (which is awesomely cool because it has USB-MIDI support built in) and one Nunchuck. I was able to build an instrument that can manage Ableton Live scenes (previous/next, launch), using one Nunchuck.

While working on the next iteration of the controller I ran into one serious roadblock. The nunchuck is designed to plug into the WiiMote game controller's single accessory port. The nunchuck wasn't designed in a way that allows two of them to be hooked up at the same time. This is because each device on an i2c bus needs to have a separate identifier, but all nunchucks have the same i2c id.

To solve this problem, I switched to the Teensy 3.0, which has two sets of i2c pins. Although it looks like the ARM Cortex M3 that the Teensy 3.0 is based on has two separate i2c buses, there is only one i2c block on the chip. Still, with a bit of creative programming, you can make things work by configuring one set of i2c pins, polling one nunchuck, then configuring the other set of pins and polling the other nunchuck. Thanks to Brian in the PJRC forum for his new i2c library and also for the help in figuring out how to make both i2c "buses" work.

Another option would have been to stick with the Teensy 2.0 and use something like a 4053-series analog multiplexer to let the Teensy talk to both nunchucks. At some point I'll adapt this instrument so it can work with a plain Arduino, and I'll include the multiplexer in that design.

How It Works

First, an explanatory video:



The code, which is available on github, works as follows:

Each time through the loop() method, we do the following

• Poll each nunchuck and record the joystick position, button state, and the current pitch (up/down) and roll (side-to-side) position.
• Record the current analog value read from the breath sensor

If the breath value has increased above a threshold value, we send one or more MIDI note on events to turn on notes, and if the value has fallen below threshold, we turn off all notes that are currently playing. This allows the player to articulate notes just like on a wind instrument.

If notes are currently sounding, we also send new MIDI continuous controller values for breath controller and also send updated aftertouch values. We send both because some synthesizer patches respond to breath controller input, while others, such as the Korg M1 software synth I use in the video, only respond to aftertouch values.

When it's time to send note on messages, we first calculate a "base" note (not a "bass" note), by looking at the "compass position" of the right joystick, and map those positions to the notes of a pentatonic scale:

Center: C6
North: D6
Northeast: F6
East: G6
Southeast: A6
South: C7
Southwest: D7
West: F7
Northwest: G7

The left joystick can be used to transpose the base note by an octave up or down by moving it in the X axis. If pushed to the left, the base note is transposed down an octave, by subtracting 12 from the base MIDI note value. If pushed to the right, the base note is transposed up an octave by adding 12 to the base MIDI note value (centered leaves the base note as is).

Once the base note has been calculated, a harmonization of the base note is created. There are several different harmonization algorithms, which can be cycled through by using the buttons on the left joystick. In the code, we use a function pointer to select which harmonization function gets called for note on events.

There are several parallel harmonization algorithms, as well as two that create slash chord voicings by choosing a bass note randomly from one of several choices.

Then, we decide how many notes of the harmonization to turn on by examining the roll value of the left nunchuck. When rolled all the way to the left, only the base note sounds. Rolling the nunchuck to the right enables more and more of the notes of the harmonization.

One final thing we do on each loop is look for button presses on the right nunchuck, and map those to MIDI note on messages on a second MIDI channel. If the upper button is pressed, we send MIDI note on and note off messages for MIDI note C0. If the lower button is pressed, we do the same, but with note C#0, and if both buttons are pressed and released, we send a MIDI D0 note on/off pair. This is intended for use with Ableton live to navigate up and down the list of scenes with the up and down buttons, and to launch the currently selected scene by pressing both buttons.

The Sounds

The sounds that you're hearing from the Blowchucks controller are coming from a Korg M1 Legacy software synthesizer, running inside Ableton Live 8 on my MacBook. The patches were programmed by Matt Traum of Patchman Music, and were developed from the ground up to work well with wind-based controllers like the Blowchucks, or commercially available wind controllers.

Summary

The Blowchucks controller is a fun instrument to play, especially when it is driving synthesizer patches that are breath-friendly. It should be really simple to build for about $70, and I hope that people reading this post will build the controller and dream up all sorts of crazy new ways to play it!

Wii Nunchuck and Ableton Live Scene Control

I've been playing around with some cheap knock-off Wii Nunchuck controllers I got from eBay. I wrote some code that allows the Nunchuck to be used for scene navigation and launch in Ableton Live.

Scene up/down is controlled by the two buttons, and to launch the currently selected scene, shake the controller.

The joystick is not used at this time. This will eventually be part of a more complicated controller, probably using two Nunchucks and a breath controller.

There's nothing too fancy going on here. Each button press sends a different MIDI note on/off pair on MIDI channel two, and a sudden change in the accelerometer readings sends another MIDI note on/off pair. Ableton's MIDI map mode makes it very easy to assign those messages to the scene controls.

Here's the code (requires a PJRC Teensy Microcontroller).

#include "Wire.h"
#include "WiiChuck.h"

// The MIDI channel we use for Ableton Live control
#define SCENE_MGMT_MIDI_CHANNEL 2
// The MIDI note for Live scene launch
#define SCENE_LAUNCH_MIDI_NOTE 0
// The MIDI note for the Live previous scene action
#define SCENE_PREV_MIDI_NOTE 1
// The MIDI note for the Live next scene action
#define SCENE_NEXT_MIDI_NOTE 2
// Suppress multiple scene lanches if < 100 ms apart
#define SCENE_LAUNCH_DELAY 100

// We need a change in Z-axis acceleration larger
// than this to launch the selected scene. You should
// adjust this value to fine-tune how hard you have
// to jerk the nunchuck to launch a scene.
#define Z_AXIS_ACCEL_THRESHOLD 500

WiiChuck chuck = WiiChuck(); // The nunchuck controller

int xVal, yVal, zVal, zSum, zAvg;
int zValues[10] = {0};
byte note;
boolean cButton, zButton;
int i;
unsigned long sceneChangeTime;

void setup() {
  // Initialize the nunchuck-related things
  cButton = zButton = false;
  chuck.begin();
}


void loop() {
  // Process nunchuck data
  chuck.update(); 
  delay(1);
  
  // Deal with accelerometer. Compute the average accelerometer
  // value over the last 10 samples. I've unfortunately forgotten
  // what internet reference lead me to using smoothed values
  // for this - apologies.
  zVal = chuck.readAccelZ();
  zSum -= zValues[i];
  zSum += zVal;
  zValues[i] = zVal;
  i = (i + 1) % 10;
  zAvg = zSum / 10;
  
  // If the average value is above some configured threshold, treat
  // the gesture as a scene launch
  if (zAvg > Z_AXIS_ACCEL_THRESHOLD) {
    if (millis() - sceneChangeTime > SCENE_LAUNCH_DELAY) {
      usbMIDI.sendNoteOn(SCENE_LAUNCH_MIDI_NOTE, 100, SCENE_MGMT_MIDI_CHANNEL);
      usbMIDI.sendNoteOff(SCENE_LAUNCH_MIDI_NOTE, 100, SCENE_MGMT_MIDI_CHANNEL);
      sceneChangeTime = millis();
    }
  } 
  
  if (chuck.cPressed()) {
    if (!cButton) {
      // Rising edge - launch current scene
      usbMIDI.sendNoteOn(SCENE_PREV_MIDI_NOTE, 100, SCENE_MGMT_MIDI_CHANNEL);
      usbMIDI.sendNoteOff(SCENE_PREV_MIDI_NOTE, 100, SCENE_MGMT_MIDI_CHANNEL);
      cButton = true;
    }
  } else {
    cButton = false;
  }
  if (chuck.zPressed()) {
    if (!zButton) {
      // Rising edge - select the next scene
      usbMIDI.sendNoteOn(SCENE_NEXT_MIDI_NOTE, 100, SCENE_MGMT_MIDI_CHANNEL);
      usbMIDI.sendNoteOff(SCENE_NEXT_MIDI_NOTE, 100, SCENE_MGMT_MIDI_CHANNEL);
      zButton = true;
    }
  } else {
    zButton = false;
  }
}

To set up Ableton Live, you need to enter MIDI mode (Command-M on a Mac), select the control to configure, then either press one of the buttons on the nunchuck or shake it, depending on which scene control you are configuring).

Depending on which Wii Nunchuck you have (knockoff or genuine Nintendo), you may need a different initialization sequence. Here's the WiiChuck.h I cobbled together from various internet sources (but most of the work comes from Tim Hirzel and Tod E. Kurt of ThingM):

/*
 * Nunchuck -- Use a Wii Nunchuck
 * Tim Hirzel http://www.growdown.com
 * 
 notes on Wii Nunchuck Behavior.
 This library provides an improved derivation of rotation angles from the nunchuck accelerometer data.
 The biggest different over existing libraries (that I know of ) is the full 360 degrees of Roll data
 from teh combination of the x and z axis accelerometer data using the math library atan2. 

 It is accurate with 360 degrees of roll (rotation around axis coming out of the c button, the front of the wii),
 and about 180 degrees of pitch (rotation about the axis coming out of the side of the wii).  (read more below)

 In terms of mapping the wii position to angles, its important to note that while the Nunchuck
 sense Pitch, and Roll, it does not sense Yaw, or the compass direction.  This creates an important
 disparity where the nunchuck only works within one hemisphere.  At a result, when the pitch values are 
 less than about 10, and greater than about 170, the Roll data gets very unstable.  essentially, the roll
 data flips over 180 degrees very quickly.   To understand this property better, rotate the wii around the
 axis of the joystick.  You see the sensor data stays constant (with noise).  Because of this, it cant know
 the difference between arriving upside via 180 degree Roll, or 180 degree pitch.  It just assumes its always
 180 roll.


 * 
 * This file is an adaptation of the code by these authors:
 * Tod E. Kurt, http://todbot.com/blog/
 *
 * The Wii Nunchuck reading code is taken from Windmeadow Labs
 * http://www.windmeadow.com/node/42
 * 
 * Conversion to Arduino 1.0 by Danjovic
 * http://hotbit.blogspot.com 
 * 
 */

#ifndef WiiChuck_h
#define WiiChuck_h


#include "Arduino.h"
#include <Wire.h>
#include <math.h>


// these may need to be adjusted for each nunchuck for calibration
#define ZEROX 510  
#define ZEROY 490
#define ZEROZ 460
#define RADIUS 210  // probably pretty universal

#define DEFAULT_ZERO_JOY_X 124
#define DEFAULT_ZERO_JOY_Y 132



class WiiChuck {
    private:
        uint8_t cnt;
        uint8_t status[6];  // array to store wiichuck output
        uint8_t averageCounter;
        //int accelArray[3][AVERAGE_N];  // X,Y,Z
        int i;
        int total;
        uint8_t zeroJoyX;   // these are about where mine are
        uint8_t zeroJoyY; // use calibrateJoy when the stick is at zero to correct
        int lastJoyX;
        int lastJoyY;
        int angles[3];

        bool lastZ, lastC;


    public:

        uint8_t joyX;
        uint8_t joyY;
        bool buttonZ;
        bool buttonC;
        void begin() 
        {
            Wire.begin();
            cnt = 0;
            averageCounter = 0;
            // instead of the common 0x40 -> 0x00 initialization, we
            // use 0xF0 -> 0x55 followed by 0xFB -> 0x00.
            // this lets us use 3rd party nunchucks (like cheap $4 ebay ones)
            // while still letting us use official oness.
            // only side effect is that we no longer need to decode bytes in _nunchuk_decode_byte
            // see http://www.arduino.cc/cgi-bin/yabb2/YaBB.pl?num=1264805255
            //  
            Wire.beginTransmission(0x52); // device address
            Wire.write(0xF0);
            Wire.write(0x55);
            Wire.endTransmission();
  
            delay(1);
            Wire.beginTransmission(0x52);
            Wire.write(0xFB);
            Wire.write((uint8_t)0x00);
            

            Wire.endTransmission();
            update();            
            for (i = 0; i<3;i++) {
                angles[i] = 0;
            }
            zeroJoyX = DEFAULT_ZERO_JOY_X;
            zeroJoyY = DEFAULT_ZERO_JOY_Y;
        }


        void calibrateJoy() {
            zeroJoyX = joyX;
            zeroJoyY = joyY;
        }

        void update() {

            Wire.requestFrom (0x52, 6); // request data from nunchuck
            while (Wire.available ()) {
                // receive byte as an integer
                status[cnt] = _nunchuk_decode_byte (Wire.read()); //
                cnt++;
            }
            if (cnt > 5) {
                lastZ = buttonZ;
                lastC = buttonC;
                lastJoyX = readJoyX();
                lastJoyY = readJoyY();
                //averageCounter ++;
                //if (averageCounter >= AVERAGE_N)
                //    averageCounter = 0;

                cnt = 0;
                joyX = (status[0]);
                joyY = (status[1]);
                for (i = 0; i < 3; i++) 
                    //accelArray[i][averageCounter] = ((int)status[i+2] << 2) + ((status[5] & (B00000011 << ((i+1)*2) ) >> ((i+1)*2))); 
                    angles[i] = (status[i+2] << 2) + ((status[5] & (B00000011 << ((i+1)*2) ) >> ((i+1)*2))); 

                //accelYArray[averageCounter] = ((int)status[3] << 2) + ((status[5] & B00110000) >> 4); 
                //accelZArray[averageCounter] = ((int)status[4] << 2) + ((status[5] & B11000000) >> 6); 

                buttonZ = !( status[5] & B00000001);
                buttonC = !((status[5] & B00000010) >> 1);
                _send_zero(); // send the request for next bytes

            }
        }


    // UNCOMMENT FOR DEBUGGING
    //byte * getStatus() {
    //    return status;
    //}

    float readAccelX() {
       // total = 0; // accelArray[xyz][averageCounter] * FAST_WEIGHT;
        return (float)angles[0] - ZEROX;
    }
    float readAccelY() {
        // total = 0; // accelArray[xyz][averageCounter] * FAST_WEIGHT;
        return (float)angles[1] - ZEROY;
    }
    float readAccelZ() {
        // total = 0; // accelArray[xyz][averageCounter] * FAST_WEIGHT;
        return (float)angles[2] - ZEROZ;
    }

    bool zPressed() {
        return (buttonZ && ! lastZ);
    }
    bool cPressed() {
        return (buttonC && ! lastC);
    }

    // for using the joystick like a directional button
    bool rightJoy(int thresh=60) {
        return (readJoyX() > thresh and lastJoyX <= thresh);
    }

    // for using the joystick like a directional button
    bool leftJoy(int thresh=60) {
        return (readJoyX() < -thresh and lastJoyX >= -thresh);
    }


    int readJoyX() {
        return (int) joyX - zeroJoyX;
    }

    int readJoyY() {
        return (int)joyY - zeroJoyY;
    }


    // R, the radius, generally hovers around 210 (at least it does with mine)
   // int R() {
   //     return sqrt(readAccelX() * readAccelX() +readAccelY() * readAccelY() + readAccelZ() * readAccelZ());  
   // }


    // returns roll degrees
    int readRoll() {
        return (int)(atan2(readAccelX(),readAccelZ())/ M_PI * 180.0);
    }

    // returns pitch in degrees
    int readPitch() {        
        return (int) (acos(readAccelY()/RADIUS)/ M_PI * 180.0);  // optionally swap 'RADIUS' for 'R()'
    }

    private:
        uint8_t _nunchuk_decode_byte (uint8_t x)
        {
            //decode is only necessary with certain initializations 
            //x = (x ^ 0x17) + 0x17;
            return x;
        }

        void _send_zero()
        {
            Wire.beginTransmission (0x52); // transmit to device 0x52
            Wire.write ((uint8_t)0x00);  // sends one byte
            Wire.endTransmission (); // stop transmitting
        }

};


#endif

Expression, Part 2 (MIDI Aftertouch)

In my previous post I showed how to map the breath pressure measured at the time a note commences to MIDI note on velocity. This allows a wind controller to play synth patches that mimic percussive or plucked instruments in a fairly natural way - the harder you blow, the harder the attack sounds (assuming that the synthesizer patch is responsive to note on velocity).

Another expression parameter present in the MIDI spec is aftertouch, sometimes referred to as pressure. After pressing a key down, a player can "lean on" the note and the keyboard will send additional MIDI messages that tell a synthesizer how hard the key is being pressed. Synth patches that respond to aftertouch will often alter a filter frequency as the pressure on the key increases (usually brightening the sound), or apply more vibrato to the note, but that's just one of many possible things that can be altered in real time.

There are actually two types of aftertouch - monophonic aftertouch, sometimes called channel pressure, and polyphonic aftertouch. With monophonic aftertouch, the keyboard only sends one value for aftertouch, no matter how many keys are being held down. With polyphonic aftertouch, the keyboard sends one aftertouch value for each key being pressed. Since a wind instrument is inherently a monophonic (one note at a time) instrument, we'll only worry about monophonic aftertouch in this post.

This sketch builds on the sketch in the MIDI note on velocity sketch (and uses the same circuit built in this post), and brings back a couple of the concepts from the breath controller post. Notice how we keep track of the last time we sent an aftertouch value, and only send a new value if a specific amount of time has passed, to avoid sending unnecessary MIDI data.

I've highlighted in yellow the additions the sketch from the previous post (so anything not in yellow is unchanged). There really isn't a lot of new code, mostly because we only need to worry about sending aftertouch values when we know a note is on.

#define MIDI_CHANNEL 1
// The threshold level for sending a note on event. If the
// sensor is producing a level above this, we should be sounding
// a note.
#define NOTE_ON_THRESHOLD 80
// The maximum raw pressure value you can generate by
// blowing into the tube.
#define MAX_PRESSURE 500

// The three states of our state machine
// No note is sounding
#define NOTE_OFF 1
// We've observed a transition from below to above the
// threshold value. We wait a while to see how fast the
// breath velocity is increasing
#define RISE_WAIT 2
// A note is sounding
#define NOTE_ON 3
// Send aftertouch data no more than every AT_INTERVAL
// milliseconds
#define AT_INTERVAL 70 

// We wait for 10 milliseconds of continuous breath

// pressure above NOTE+ON_THRESHOLD before we turn on

// the note, to de-glitch

#define RISE_TIME 10 

// The five notes, from which we choose one at random
unsigned int notes[5] = {60, 62, 65, 67, 69};

// We keep track of which note is sounding, so we know
// which note to turn off when breath stops.
int noteSounding;
// The value read from the sensor
int sensorValue;
// The state of our state machine
int state;
// The time that we noticed the breath off -> on transition
unsigned long breath_on_time = 0L;
// The breath value at the time we observed the transition
int initial_breath_value;
// The aftertouch value we will send
int atVal;
// The last time we sent an aftertouch value
unsigned long atSendTime = 0L;


void setup() {
  state = NOTE_OFF;  // initialize state machine
}

int get_note() {
  return notes[random(0,4)];
}

int get_velocity(int initial, int final, unsigned long time_delta) {
  return map(final, NOTE_ON_THRESHOLD, MAX_PRESSURE, 0, 127);
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  if (state == NOTE_OFF) {
    if (sensorValue > NOTE_ON_THRESHOLD) {
      // Value has risen above threshold. Move to the RISE_TIME
      // state. Record time and initial breath value.
      breath_on_time = millis();
      initial_breath_value = sensorValue;
      state = RISE_WAIT;  // Go to next state
    }
  } else if (state == RISE_WAIT) {
    if (sensorValue > NOTE_ON_THRESHOLD) {
      // Has enough time passed for us to collect our second
      // sample?
      if (millis() - breath_on_time > RISE_TIME) {
        // Yes, so calculate MIDI note and velocity, then send a note on event
        noteSounding = get_note();
        int velocity = get_velocity(initial_breath_value, sensorValue, RISE_TIME);
        usbMIDI.sendNoteOn(noteSounding, velocity, MIDI_CHANNEL);
        state = NOTE_ON;
      }
    } else {
      // Value fell below threshold before RISE_TIME passed. Return to
      // NOTE_OFF state (e.g. we're ignoring a short blip of breath)
      state = NOTE_OFF;
    }
  } else if (state == NOTE_ON) {
    if (sensorValue < NOTE_ON_THRESHOLD) {
      // Value has fallen below threshold - turn the note off
      usbMIDI.sendNoteOff(noteSounding, 100, MIDI_CHANNEL);  
      state = NOTE_OFF;
    } else {
      // Is it time to send more aftertouch data?
      if (millis() - atSendTime > AT_INTERVAL) {
        // Map the sensor value to the aftertouch range 0-127
        atVal = map(sensorValue, NOTE_ON_THRESHOLD, 1023, 0, 127);
        usbMIDI.sendAfterTouch(atVal, MIDI_CHANNEL);
        atSendTime = millis();
      }
    }
  }
}

Here are a couple of recordings I made. In the first, I play some long notes that have a hard initial attack, then I immediately back off the breath, and blow harder and harder until I release the note (a fp plus a crescendo for you musicians).

In this first example, the synth is not responding to aftertouch, so you just hear a static note.

In the second example, I've modified the synth patch to apply vibrato to the patch, and the amount of vibrato depends on the aftertouch value.

Are you able to hear the vibrato (wavering sound) at the end of the notes?

What's Next

With the ability to add expressiveness via MIDI note velocity and aftertouch, we can make some pretty interesting sounds, but we're still mimicking a keyboard instrument. To realize the full potential of an electronic wind instrument, we're going to need to get synthesizers to respond to us in a way that makes sense for the way that a wind instrument works.

While I am by no means an expert in synth programming, I'll share what I know, and maybe we can come up with some interesting sounds.

Expression, part 1 (MIDI Note On Velocity)

In our previous Arduino sketches, we created musical instruments that were very limited in terms of expression. In this post, I'll discuss two ways that we can give the instrument more expressive capabilities that utilize the performer's breath - mapping breath articulation to MIDI note on velocity, and mapping breath values to MIDI aftertouch after the initial attack.

These two techniques are useful if you are trying to play an existing synthesizer patch that is not specifically designed for wind control. They still leave a lot to be desired if you want to create an instrument that plays like an acoustic wind instrument, but I'll cover that topic in a later post.

An Inventory Of Expression Parameters

Before I go much further, let's take a moment to understand the various ways the the MIDI spec conveys expressive data, and how that MIDI data relates to a breath sensor.

Note On Velocity

With a velocity-sensitive keyboard, the MIDI note on velocity describes how hard the key on the keyboard was pressed (strictly speaking, it's the speed with which the finger moves the piano key through its range of motion - that's why it's called "velocity" and not "force"). For percussive instruments like drums or keyboards, it usually controls the overall volume of the ADSR envelope. Note on values range from 0 to 127, with 0 representing the lightest possible touch, and 127 representing the maximum force.

For a wind controller, it's not so clear. When a wind player articulates a note, the note starts from zero breath and increases at some rate controlled by the player. In the case where the player is using a hard articulation (that is, one with a fast rise time), it may make sense to try to map the rise time of the sensor to a note on velocity. This can be helpful if you are trying to play a patch that behaves like a percussive or plucked instrument. We'll implement this in the first sketch in this series, and see how it works out. As a hint, it's kind of a dead-end if we really want to build an instrument that responds like a real-world wind instrument, but the concepts here are generally useful, plus if you really do want to play a piano or drum patch, it'll help you play that patch with more expression.

There is also a corresponding note off velocity. I don't think this is widely used in synth patches, but it would control the duration of the R phase of the ADSR envelope. We won't implement it in our instrument's code, to keep things simple.

Aftertouch

In addition to detecting how hard a key was initially pressed, many MIDI keyboards can detect how hard a key is being pressed while it it being held down and transmit that data via MIDI aftertouch messages. With many synth patches, pressing harder on the key will engage a vibrato effect like the modulation wheel.

With our wind controller, It's pretty simple to send aftertouch data that corresponds to the breath values. So, if a patch is configured to respond to aftertouch, we should be able to make it respond to how hard we blow after starting the note. We'll implement that in our sketch and see how it affects a few different synthesizer patches.

Continuous Controllers

We saw how continuous controllers work in a previous post. They are MIDI messages that aren't associated with any particular note. There are many different continuous controllers defined (here's a complete list).  Some synthesizers, and especially software-based synthesizers, allow you to route any continuous controller to any synthesizer parameter. When we get to the post about designing synth patches for wind control, we'll see how powerful this capability is.

Mapping Breath to Note On Velocity

 Here's a sketch that will allow you to play MIDI patches that respond to note on velocity. Most piano, drum, and guitar patches will probably respond to note on velocity information.

The basic approach is to take the reading from the breath sensor at the time the note begins, and map that to a value in the range 0 to 127 (the minimum and maximum allowable MIDI note on velocity values), and send that with the note on event. We can use the Arduino map() method to do that math, like we did in our continuous controller sketch.

The only slightly tricky thing is that, when the breath starts, it takes a few milliseconds to build to its final value, so if we use the first value we read, it won't be as "loud" as the player intended. To get around that, we actually wait a little while after we see the breath go above the note on threshold value, and then re-sample the pressure sensor. The second sample is the one we map to the velocity.

The other concept in this sketch that you may not be familiar with it the use of a very simple finite state machine. While it may sound complicated, it's actually pretty simple. At any time, the sketch is in one of three states:

• Note Off State - no note is sounding
• Rise Time State - the performer has started to blow into the sensor, and we're waiting a bit for the pressure to rise to its final value
• Note On State - a note is sounding

 For each state in a finite state machine, you also have to know what actions will move you from that state to some other state. One way to do that is via a table:


Current State   Input                       Next State
======================================================
NOTE_OFF        Breath value goes above     RISE_TIME
                NOTE_ON_THRESHOLD 

RISE_TIME       RISE_TIME milliseconds      NOTE_ON
                have elapsed

RISE_TIME       Breath value goes below     NOTE_OFF
                NOTE_ON_THRESHOLD 

NOTE_ON         Breath value goes below     NOTE_OFF 
                NOTE_ON_THRESHOLD
  

Another way to visualize a state machine is with a state graph. The states are shown as ovals, and the transitions between states are the edges (curved arrows) in the graph. Note that two of the states have transitions that point to themselves (I didn't include those in the table above). That's perfectly ok and shows that some inputs might not cause a transition out of the current state. Click the diagram to see a larger version if it's hard to read.

Here's the sketch: 

#define MIDI_CHANNEL 1
// The threshold level for sending a note on event. If the
// sensor is producing a level above this, we should be sounding
// a note.
#define NOTE_ON_THRESHOLD 80
// The maximum raw pressure value you can generate by
// blowing into the tube.
#define MAX_PRESSURE 500

// The three states of our state machine
// No note is sounding
#define NOTE_OFF 1
// We've observed a transition from below to above the
// threshold value. We wait a while to see how fast the
// breath velocity is increasing
#define RISE_TIME 10
// A note is sounding
#define NOTE_ON 3

// The five notes, from which we choose one at random
unsigned int notes[5] = {60, 62, 65, 67, 69};

// We keep track of which note is sounding, so we know
// which note to turn off when breath stops.
int noteSounding;
// The value read from the sensor
int sensorValue;
// The state of our state machine
int state;
// The time that we noticed the breath off -> on transition
unsigned long breath_on_time = 0L;
// The breath value at the time we observed the transition
int initial_breath_value;

void setup() {
  state = NOTE_OFF;  // initialize state machine
}

int get_note() {
  return notes[random(0,4)];
}

int get_velocity(int initial, int final, unsigned long time_delta) {
  return map(final, NOTE_ON_THRESHOLD, MAX_PRESSURE, 0, 127);
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  if (state == NOTE_OFF) {
    if (sensorValue > NOTE_ON_THRESHOLD) {
      // Value has risen above threshold. Move to the RISE_TIME
      // state. Record time and initial breath value.
      breath_on_time = millis();
      initial_breath_value = sensorValue;
      state = RISE_TIME;  // Go to next state
    }
  } else if (state == RISE_TIME) {
    if (sensorValue > NOTE_ON_THRESHOLD) {
      // Has enough time passed for us to collect our second
      // sample?
      if (millis() - breath_on_time > RISE_TIME) {
        // Yes, so calculate MIDI note and velocity, then send a note on event
        noteSounding = get_note();
        int velocity = get_velocity(initial_breath_value, sensorValue, RISE_TIME);
        usbMIDI.sendNoteOn(noteSounding, velocity, MIDI_CHANNEL);
        state = NOTE_ON;
      }
    } else {
      // Value fell below threshold before RISE_TIME passed. Return to
      // NOTE_OFF state (e.g. we're ignoring a short blip of breath)
      state = NOTE_OFF;
    }
  } else if (state == NOTE_ON) {
    if (sensorValue < NOTE_ON_THRESHOLD) {
      // Value has fallen below threshold - turn the note off
      usbMIDI.sendNoteOff(noteSounding, 100, MIDI_CHANNEL);  
      state = NOTE_OFF;
    }
  }
}

Here's how it sounds. I play four notes quietly, four notes loud, four more notes quietly, and then do four notes of increasing volume (a crescendo, in musical terms), and eight notes of decreasing volume (a diminuendo). The sketch is randomly picking which notes to play, but the attack velocity of each note is under my control.

Looking at the MIDI data, it looks like the values are ranging from a low of about 4 to a maximum of about 100, so I could alter the MAX_PRESSURE value in the sketch to fix that. But you get the idea.

Here's another rhythmic motif in 9/8 that I played on a different wind controller, a Yamaha WX-7, that also maps breath to note on velocity. Because of this mapping, I'm able to accent the 1st, 4th, 6th, and 8th notes of each group of nine, resulting in a 3 + 2 + 2 + 2 pattern.


In part two, I'll add MIDI aftertouch to this sketch.

Breath and Note On/Off Transitions

If we're going to build a MIDI wind instrument controller, it's going to need to send MIDI note on and off messages - it's just part of the MIDI spec. But the tricky part is knowing when your breath is producing enough pressure to justify sending the MIDI note on message.

An approach that makes sense for our project is to choose some value a little above the "idle" value that the pressure sensor reads (that is, when you're not blowing into it). My sensor produces values of about 65 when I'm not blowing into it, and a threshold value of 100 seems to work well. In the Arduino sketches in this post, you may need to change the threshold value to something appropriate for the way your air tubing is set up, as well as your playing style. To adjust that threshold, look for the line:

#define NOTE_ON_THRESHOLD 100

And experiment with different values until you find something that works. Don't forget to recompile and upload the sketch after you make changes.

One Note

The following sketch reads the pressure sensor and sends a MIDI Note On event for middle C (the C about in the middle of the piano keyboard) when it detects that the pressure is above the threshold value. It sends a MIDI Note Off event when the pressure is below the threshold value. It also keeps track of whether a note is sounding or not, so it knows if a Note On/Off message needs to be sent (if the note is already on, we don't need, or want, to send another Note On message).

#define MIDI_CHANNEL 1
// For this sketch, we only can play one note
#define MIDI_NOTE 60 // Middle C (C4)
// The threshold level for sending a note on event. If the
// sensor is producing a level above this, we are in note on
// state, otherwise note off
#define NOTE_ON_THRESHOLD 100

// We keep track of whether a note is sounding or not,
// so we know whether to send a note on or off event.
boolean noteSounding = false;
// The value read from the sensor
int sensorValue;

void setup() {
  // Nothing to initialize for this sketch
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  // Send the appropriate MIDI note on or off message
  if (sensorValue > NOTE_ON_THRESHOLD) {
    if (noteSounding) {
      // Nothing to do - note is already on
    } else {
      // Value has risen above threshold - turn the note on
      usbMIDI.sendNoteOn(MIDI_NOTE, 100, MIDI_CHANNEL);
      noteSounding = true;
    }
  } else {
    if (noteSounding) {
      // Value has fallen below threshold - turn the note off
      usbMIDI.sendNoteOff(MIDI_NOTE, 100, MIDI_CHANNEL);
      noteSounding = false;
    } else {
      // Nothing to do - note is already off
    }
  }
  // Delay a bit to avoid glitches (very short notes that
  // sound because the breath falls below the threshold
  // value, then goes back above it). Adding 20
  // milleseconds of delay reduces the responsiveness
  // of the instrument, and we'll discuss better ways
  // of handling this in later posts. For now, this is
  // good enough.
  delay(20);
}

Here's how the sketch sounds when playing the Grand Piano patch from GarageBand.

Four long notes:

A synth pad, long notes:

And to test the responsiveness, here I'm playing faster and faster notes (using a technique called double-tonguing and then flutter-tonguing). It keeps up pretty well:


Ok, great. We can send MIDI Note On/Off messages. But this is pretty boring, only playing one note. We'll be talking about how to select notes based on fingerings in later posts, but let's at least make our sketch choose a couple of different notes at random.

Here's a modification to the previous sketch. Instead of always sending a middle C for the note, whenever a Note On event is to be sent, it randomly chooses one of 5 notes from a pentatonic scale by using the random() method from the Arduino library. Look at the get_note() method - that's where we pick the random note to play. The Arduino library's random() method picks a number from 0 through 4 for us, and we use that to select one of the MIDI notes 60 (middle C), 62 (D), 65 (F), 67 (G), or 69 (A).

#define MIDI_CHANNEL 1
// The threshold level for sending a note on event. If the
// sensor is producing a level above this, we are in note on
// state, otherwise note off
#define NOTE_ON_THRESHOLD 80  

unsigned int notes[5] = {60, 62, 65, 67, 69};

// We keep track of which note is sounding. The value
// -1 means no note is sounding.
int noteSounding = -1;
// The value read from the sensor
int sensorValue;

void setup() {
  // Nothing to initialize for this sketch
}

int get_note() {
   return notes[random(0,4)];
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  // Send the appropriate MIDI note on or off message
  if (sensorValue > NOTE_ON_THRESHOLD) {
    if (noteSounding != -1) {
      // Nothing to do - note is already on
    } else {
      // Value has risen above threshold - turn the note on
      noteSounding = get_note();
      usbMIDI.sendNoteOn(noteSounding, 100, MIDI_CHANNEL);
    }
  } else {
    if (noteSounding != -1) {
      // Value has fallen below threshold - turn the note off
      usbMIDI.sendNoteOff(noteSounding, 100, MIDI_CHANNEL);
      noteSounding = -1;
    } else {
      // Nothing to do - note is already off
    }
  }
  // Delay a bit to avoid glitches (very short notes that
  // sound because the breath falls below the threshold
  // value, then goes back above it). Adding 20
  // milleseconds of delay reduces the responsiveness
  // of the instrument, and we'll discuss better ways
  // of handling this in later posts. For now, this is
  // good enough.
  delay(20);
}

Here's how it sounds. I'm playing a syncopated rhythm, using the GarageBand Grand Piano patch. Notice how the note pitches change randomly:

Here I'm playing the new sketch and using the "Kotu Chords" patch from GarageBand's Synth Textures collection, but playing the same rhythm. This patch harmonizes the input notes, so even though the get_note() method only produces one randomly selected note, once I feed that note into the synthesizer, I get about 4 notes out.


A cool aspect of this sketch is that, since we're using the random() method from the Arduino library to choose which note to play, two performances of the same piece on this instrument are likely to be very different. And by combining our pretty simple breath based controller with some interesting patches from GarageBand, we're able to make some interesting sounds. Try playing around with different patches in GarageBand, or if you're using a different audio environment and/or OS, experiment with what you have. Upload your creations and share links to them here.

Next

While these sketches can produce some interesting sounds, they are pretty much devoid of expression. All the notes are exactly the same volume, and we have no control over how the note evolves over time. In my next post, I'll talk about how to create an instrument that is much more responsive to the nuances of breath control.

Building a Breath Controller

In this project, I'll build on my previous post on reading breath pressure into an Arduino or Arduino clone. We'll use the same circuit from that post, and write some new code that maps the sensor values we read to a MIDI continuous controller (which I'll refer to as a CC from here on out). That CC data will, in turn, control a parameter of a software synthesizer.

What's a MIDI CC?

MIDI continuous controllers are a standard way for performers to modify the sound produced by the synthesizers they are playing. The "continuous" part of the name means that the values sent on the MIDI bus control some parameter with a range. If you've played a MIDI keyboard that has a modulation wheel:

you know that the more you turn the "mod wheel", the more vibrato you hear, at least with most common synth patches. In the following example, I play a chord, and turn the modulation wheel from zero to full on, then back to zero.

If you dig into the MIDI spec, you'll discover that the mod wheel is only one of many continuous controllers defined. Each one has a unique number assigned to it, and some have a name, like "Modulation Wheel" or "Pan". The controller name describes what type of control is typically going to generate that data. On a traditional MIDI keyboard, if you move the mod wheel, you'll generate Continuous Controller 1 data.

If you think about it, there are many real-life things that naturally can be expressed as a continuous controller: How far you have twisted a knob. How far to the left or right your are tilting your hand. How far apart your hands are. Or how hard you are blowing air. All of these things can be mapped to some number that ranges from some minimum value to some maximum value (provided you can measure them and translate them to MIDI).

The MIDI spec states that continuous controller values vary from a minimum value of zero to a maximum value of 127. This means CC values have 7 bit resolution, which, for some applications, is not acceptable - the results sound odd. I'll talk about those later, but for many of the common synthesizer parameters that are controller by CCs, 7 bits is fine.

How the Pressure Sensor Works

Before we get into wiring up and coding the project, let's dive into how the Freescale pressure sensor works. It relies on something called the Piezoresistive Effect.

There are some materials that exhibit the property that, as you press on them, the resistance through the material changes. The Freescale sensor has a small piece of this type of material inside, and when the pressure inside the case of the sensor increases, the resistance through this material changes. The sensor also has circuitry to amplify the signal and make it work consistently even if the temperature varies.

Advances in chip fabrication techniques have allowed manufacturers to make these type of sensors in very small packages. By contrast, the Yamaha WX-7, a 1980s-era wind instrument controller, has a breath sensor that is much larger than the Freescale sensor used in this project. Although I don't know how the WX-7's pressure sensor works, I suspect it's based on the same principle.

Mapping Sensor Values

We'll now build on the circuit in my previous post and generate MIDI continuous controller data based on how hard a performer is blowing into the breath sensor.

Remember that our Arduino's analog inputs convert the values they read into a value that varies between 0 and 1023. But the MIDI spec wants you to provide values between 0 and 127. What to do?

Well, the math is pretty simple. Just divide the value read from the input by 1024, and multiply it by 128. But the arduino library has a nice convenience function named map() that does exactly this. For example:

  int sensorValue1 = analogRead(A0);
  int ccVal = map(sensorValue1, 0, 1023, 0, 127);

The code snippet above will linearly map values in the range 0-1023 to values in the range 0-127.

Generating MIDI CC Values

The PJRC USB MIDI support makes sending MIDI data over USB just ridiculously simple. From the PJRC USB MIDI reference:

usbMIDI.sendControlChange(control, value, channel)

 Where "control" is the MIDI controller to change, "value" is the mapped sensor value, and "channel" is the MIDI channel on which to send the data. Although we haven't talked about MIDI channels yet, for these exercises we'll send the data on MIDI channel 1 and then make sure that our MIDI synthesizers are either set up to listen on channel 1 or are set to "omni" mode, in which they respond to MIDI data on any MIDI channel.

Our First, and Imperfect, Sketch

#define MOD_WHEEL_CONTROLLER 1
#define MIDI_CHANNEL 1

// The value read from the sensor
int sensorValue;
// The CC value we will send
int ccVal;

void setup() {
  // Nothing to initialize for this sketch
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  // Map the value, which may range from 0 to 1023,
  // to a value in the range 0 to 127, which is
  // the valid range for a MIDI continuous controller
  ccVal = map(sensorValue, 0, 1023, 0, 127);
  // And send the value as a MIDI CC message
  usbMIDI.sendControlChange(MOD_WHEEL_CONTROLLER, ccVal, MIDI_CHANNEL);
}

 To load this sketch, first go to the "Tools" menu and choose "USB Type" and select "MIDI". Then click on the upload button.

Once you've loaded the sketch, you'll need to load a software synthesizer host on your Mac/PC/Linux box. While I can't cover every possibility in this post, I'll cover the basic steps for running Garage Band on a Mac and getting mod wheel output from our circuit.

1. Start Garage Band.
2. If a project opens automatically, choose Close from the File menu and choose New Project...
3. Select the Keyboard Collection template and give the project a name in the dialog that appears.
4.  In the list of instruments that appears, click on Solo Star - it's an interesting-sounding synth that responds to mod wheel input.
5. Select Musical Typing from the Window menu. A window appears where you can click on a virtual MIDI keyboard.
6. If your Teensy controller isn't plugged in, plug it in now. When you plug in a MIDI-USB device, GarageBand should pop up a window informing you that the number of MIDI inputs has changed.
7. Click and hold down the mouse button on a note in the Musical Typing window. In the sound sample below, I clicked and held the "F" key.
8. You should be hearing a synthesizer playing a note now. Keep holding the mouse button and blow into the tube connected to your pressure sensor. Start blowing gently, blow harder and harder, then gradually blow less and less.
9. You should hear something about like this

And if you double-click the segment you just recorded, and select "Modulation" from the "View" menu, you can see a graph of the values the Teensy sent.

Pretty cool, huh? Actually, no, there are two serious problems with my sketch.

Shut Up, Already!

Notice that we send a MIDI CC value every time the the loop() method is called. This means that you're spamming the MIDI instruments as fast as your microcontroller can make it through its loop() method. This is not good because (a) the MIDI bus has a finite bandwidth, and (b) you can overload software synthesizers with this data, as they try to adjust internal parameters in real time.

There are two approaches I can think of to address this problem:

1. Only send a new CC value if a certain amount of time has passed, e.g. 20 milliseconds, or
2. Only send a new CC value if the new value is different from the previously sent value to be interesting to the synthesizer.

The first method is really easy to implement.  Here's a sketch that will only send CC values every 20 milliseconds.

#define MOD_WHEEL_CONTROLLER 1
#define MIDI_CHANNEL 1
// Send continuous controller message no more than
// every CC_INTERVAL milliseconds
#define CC_INTERVAL 20

// The last time we sent a CC value
unsigned long ccSendTime = 0L;
// The value read from the sensor
int sensorValue;
// The CC value we will send
int ccVal;

void setup() {
  // Nothing to initialize for this sketch
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  // Map the value, which may range from 0 to 1023,
  // to a value in the range 0 to 127, which is
  // the valid range for a MIDI continuous controller
  ccVal = map(sensorValue, 0, 1023, 0, 127);
  // And send the value as a MIDI CC message
  if (millis() - ccSendTime > CC_INTERVAL) {
      usbMIDI.sendControlChange(MOD_WHEEL_CONTROLLER, ccVal, MIDI_CHANNEL);
      ccSendTime = millis();
  }
}

Don't Send Useless Data

The other problem is that, even when you aren't blowing into the tube, it's sending data to the MIDI instruments. If no air is going through the tube, then why send any data at all?

This is pretty easy to fix. Recall that, even when no air was being blown through the tube, it produced a voltage that the analog-to-digital converter on the Arduino interpreted as about 64. So, if the value we read from the input port is less than about 70, let's just not send anything at all. Problem solved.

One subtle issue is that when the pressure falls below the threshold value, we need to  send a zero value. Otherwise, when you stop blowing, the controller would remain at the last value we sent, rather than zero. We handle this in the last else clause.

#define MOD_WHEEL_CONTROLLER 1
#define MIDI_CHANNEL 1
// Send continuous controller message no more than
// every CC_INTERVAL milliseconds
#define CC_INTERVAL 20
// Only send CC data if the pressure sensor reading
// a value larger than this.
#define BREATH_THRESHOLD 70

// The last time we sent a CC value
unsigned long ccSendTime = 0L;
// The value read from the sensor
int sensorValue;
// The CC value we will send
int ccVal;
// The last CC value we sent
int lastCcVal = 0;

void setup() {
  // Nothing to initialize for this sketch
}

void loop() {
  // Only read the sensor if enough time has passed
  if (millis() - ccSendTime > CC_INTERVAL) {
    // read the input on analog pin 0
    sensorValue = analogRead(A0);
    if (sensorValue > BREATH_THRESHOLD) {
      // Map the value, which may range from BREATH_THRESHOLD
      // to 1023, to a value in the range 0 to 127, which is
      // the valid range for a MIDI continuous controller
      ccVal = lastCcVal = map(sensorValue, BREATH_THRESHOLD, 1023, 0, 127);
      // And send the value as a MIDI CC message
      usbMIDI.sendControlChange(MOD_WHEEL_CONTROLLER, ccVal, MIDI_CHANNEL);
      ccSendTime = millis();
    } 
    else if (lastCcVal > 0) {
      // The pressure has just dropped below the threshold, so
      // send a CC value of zero
      usbMIDI.sendControlChange(MOD_WHEEL_CONTROLLER, 0, MIDI_CHANNEL);
      ccSendTime = millis();
      lastCcVal = 0;
    }
  }
}

Another thing you can do with this code is convert it to a MIDI Breath Controller and do cool stuff like in this video:

You can build that For about $36.

Next

In the next post, I'll show how to use the pressure sensor to manage MIDI Note On and Note Off events, which will allow us to articulate like a woodwind or brass player does.