Sunday, January 27, 2013

Expression, Part 2 (MIDI Aftertouch)


In my previous post I showed how to map the breath pressure measured at the time a note commences to MIDI note on velocity. This allows a wind controller to play synth patches that mimic percussive or plucked instruments in a fairly natural way - the harder you blow, the harder the attack sounds (assuming that the synthesizer patch is responsive to note on velocity).

Another expression parameter present in the MIDI spec is aftertouch, sometimes referred to as pressure. After pressing a key down, a player can "lean on" the note and the keyboard will send additional MIDI messages that tell a synthesizer how hard the key is being pressed. Synth patches that respond to aftertouch will often alter a filter frequency as the pressure on the key increases (usually brightening the sound), or apply more vibrato to the note, but that's just one of many possible things that can be altered in real time.

There are actually two types of aftertouch - monophonic aftertouch, sometimes called channel pressure, and polyphonic aftertouch. With monophonic aftertouch, the keyboard only sends one value for aftertouch, no matter how many keys are being held down. With polyphonic aftertouch, the keyboard sends one aftertouch value for each key being pressed. Since a wind instrument is inherently a monophonic (one note at a time) instrument, we'll only worry about monophonic aftertouch in this post.

This sketch builds on the sketch in the MIDI note on velocity sketch (and uses the same circuit built in this post), and brings back a couple of the concepts from the breath controller post. Notice how we keep track of the last time we sent an aftertouch value, and only send a new value if a specific amount of time has passed, to avoid sending unnecessary MIDI data.

I've highlighted in yellow the additions the sketch from the previous post (so anything not in yellow is unchanged). There really isn't a lot of new code, mostly because we only need to worry about sending aftertouch values when we know a note is on.


#define MIDI_CHANNEL 1
// The threshold level for sending a note on event. If the
// sensor is producing a level above this, we should be sounding
// a note.
#define NOTE_ON_THRESHOLD 80
// The maximum raw pressure value you can generate by
// blowing into the tube.
#define MAX_PRESSURE 500

// The three states of our state machine
// No note is sounding
#define NOTE_OFF 1
// We've observed a transition from below to above the
// threshold value. We wait a while to see how fast the
// breath velocity is increasing
#define RISE_WAIT 2
// A note is sounding
#define NOTE_ON 3
// Send aftertouch data no more than every AT_INTERVAL
// milliseconds
#define AT_INTERVAL 70 
// We wait for 10 milliseconds of continuous breath
// pressure above NOTE+ON_THRESHOLD before we turn on
// the note, to de-glitch
#define RISE_TIME 10 
// The five notes, from which we choose one at random
unsigned int notes[5] = {60, 62, 65, 67, 69};

// We keep track of which note is sounding, so we know
// which note to turn off when breath stops.
int noteSounding;
// The value read from the sensor
int sensorValue;
// The state of our state machine
int state;
// The time that we noticed the breath off -> on transition
unsigned long breath_on_time = 0L;
// The breath value at the time we observed the transition
int initial_breath_value;
// The aftertouch value we will send
int atVal;
// The last time we sent an aftertouch value
unsigned long atSendTime = 0L;


void setup() {
  state = NOTE_OFF;  // initialize state machine
}

int get_note() {
  return notes[random(0,4)];
}

int get_velocity(int initial, int final, unsigned long time_delta) {
  return map(final, NOTE_ON_THRESHOLD, MAX_PRESSURE, 0, 127);
}

void loop() {
  // read the input on analog pin 0
  sensorValue = analogRead(A0);
  if (state == NOTE_OFF) {
    if (sensorValue > NOTE_ON_THRESHOLD) {
      // Value has risen above threshold. Move to the RISE_TIME
      // state. Record time and initial breath value.
      breath_on_time = millis();
      initial_breath_value = sensorValue;
      state = RISE_WAIT;  // Go to next state
    }
  } else if (state == RISE_WAIT) {
    if (sensorValue > NOTE_ON_THRESHOLD) {
      // Has enough time passed for us to collect our second
      // sample?
      if (millis() - breath_on_time > RISE_TIME) {
        // Yes, so calculate MIDI note and velocity, then send a note on event
        noteSounding = get_note();
        int velocity = get_velocity(initial_breath_value, sensorValue, RISE_TIME);
        usbMIDI.sendNoteOn(noteSounding, velocity, MIDI_CHANNEL);
        state = NOTE_ON;
      }
    } else {
      // Value fell below threshold before RISE_TIME passed. Return to
      // NOTE_OFF state (e.g. we're ignoring a short blip of breath)
      state = NOTE_OFF;
    }
  } else if (state == NOTE_ON) {
    if (sensorValue < NOTE_ON_THRESHOLD) {
      // Value has fallen below threshold - turn the note off
      usbMIDI.sendNoteOff(noteSounding, 100, MIDI_CHANNEL);  
      state = NOTE_OFF;
    } else {
      // Is it time to send more aftertouch data?
      if (millis() - atSendTime > AT_INTERVAL) {
        // Map the sensor value to the aftertouch range 0-127
        atVal = map(sensorValue, NOTE_ON_THRESHOLD, 1023, 0, 127);
        usbMIDI.sendAfterTouch(atVal, MIDI_CHANNEL);
        atSendTime = millis();
      }
    }
  }
}


Here are a couple of recordings I made. In the first, I play some long notes that have a hard initial attack, then I immediately back off the breath, and blow harder and harder until I release the note (a fp plus a crescendo for you musicians).

In this first example, the synth is not responding to aftertouch, so you just hear a static note.

In the second example, I've modified the synth patch to apply vibrato to the patch, and the amount of vibrato depends on the aftertouch value.

Are you able to hear the vibrato (wavering sound) at the end of the notes?

What's Next


With the ability to add expressiveness via MIDI note velocity and aftertouch, we can make some pretty interesting sounds, but we're still mimicking a keyboard instrument. To realize the full potential of an electronic wind instrument, we're going to need to get synthesizers to respond to us in a way that makes sense for the way that a wind instrument works.

While I am by no means an expert in synth programming, I'll share what I know, and maybe we can come up with some interesting sounds.

18 comments:

  1. Hey Gordon,

    Can't wait to see the next of this tutorials man, the tutorials you make are great regarding wind synthesis. I hope you post soon the next update! Keep me updated please! I would seriously recommend your website to my friends from lower years to see this website for reference if they plan to make a project involving wind synthesis.

    I downloaded a software synthesizer that is very similar to garageband because im using Windows and not a mac. Waiting for the next of your tutorials so that I can go to the next step but they have been perfectly helpful for me

    Can you please include tutorial on how you let your button combinations from the arduino to communicate with the Garageband to play a certain note. Thanks!
    -Ryan

    ReplyDelete
  2. When I did my wind controller, I used value string for keeping CC values, instead of delays, for not flooding channel with the same CC values.

    When CC message is sent to computer, it's also stored in the string LAST_CC (for example).
    When the next loop runs, before sending CC from the breath controller, it's first compared to LAST_CC. If values are different, CC is sending to computer, and it's value writes in LAST_CC. Otherwise, nothing is sending.

    Hope it helps.

    ReplyDelete
  3. That is good advice, Alexander. I think, though, that it's still a good idea to limit the rate of CC messages even if you keep track of the last CC value sent. Imagine a continuous controller that is very "jittery" (like a pressure sensor). It might be the case that the current CC value is always different from the LAST_CC value.

    One might also average the last raw values from the pressure sensor and only send a new CC message if the new calculated average value differs. That could help with a noisy input signal.

    ReplyDelete
    Replies
    1. If you want do discuss the theme, you can write me an email to fundorin@gmail.com.
      It's been a year since I've build my controller, and it seems like I'm ready to build a new one now. It'll be more simple than previous one and will allow me to play only 3 chromatic octaves, instead of four that I have now. Anyway, I found myself playing the highest octave less frequently that the other ones and many woodwind VST's doesn't even contain four octaves sampled or synthesized.

      Delete
    2. BTW, I din't have any issues with "jitter". My pressure sensor works smooth, according to CC records in Reaper.

      Delete
    3. Sorry for flooding, but I've checked my sketch, and there's a smoothing applied to pressure sensor data, which I used earlier. Don't remember why it's commented in the last script version. Maybe, I've found, that bandwith is enough to send CC continuosly or some other occasion.

      Anyway, here's how the smoothing was done:

      // sensP = 0.9 * sensP + 0.1 * analogRead(sensPpin);

      It called sensP, cause I also have sensV, which is for drawing (2way sensor).

      P.S. Those CAPTCHAs are really freaking me out!

      Delete
  4. Sorry about the captchas, Alexander - I'm cheap and am using the free Blogger.

    Thanks for sharing your experiences. If I recall correctly from reading the datasheets, some pressure sensors may implement hysteresis on the chip, so the values you see may already be smoothed. Which pressure sensor are you using?

    ReplyDelete
  5. AFAIR it is MPXV5010DP from Freescale. It have two inputs for positive and negative pressure measuring. I using the second one as a calibration at power on, or when pressing the reset button. My controller have to of the. For exhale I use pressure input from the first sensor, and for inhale is the second input of the second sensor.

    ReplyDelete
  6. Interesting dual-sensor setup. Mine only has a single sensor with a single (blow) port, therefore my instrument doesn't have an inhale mode. What do you use your inhale mode for? It occurs to me that one could use it to put the instrument into a "meta" mode, where the controls perform some alternate function.

    ReplyDelete
    Replies
    1. Sorry for mistypes.

      I used two of them sensors to reduce number of the buttons. Exhale for C and inhale for D on the same button press.
      For what you called "meta", if I understand this term right, I used a separate switch. When it was turned on, all play buttons acted like octave\transpose shift, program change +\-, favorite programs, etc. This is the mode with two extra "shift" buttons. With a switch turned on, controller acted like a mixer with play, rec, stop, pause, undo, redo, track navigation.
      So, with a combination of the switch and two shifts, I had 4 extra modes to control everything by pressing play buttons. One is just a switch, second is switch+shift1, third is switch+shift2 and fourth is switch+both shifts. Actually, I couldn't find how to use the last mode, so it was like a reserve for future needs.
      My layout was based on a chromatic harmonica, where 4 holes (exhale, inhale and shift button) give you a full chromatic octave. I just switched holes to buttons, and placed octaves one above the other. Full four octaves in one small device.
      The only reason I see in adding the second sensor to my new controller is to be able to play melodies without taking pauses for taking a breath.

      Delete
  7. Hi Gordon,

    Great tutorials. Are you going to add more MIDI notes at all? It would be great if a few octaves would be covered rather than the 5 random notes.

    Thanks very much,
    Si B.

    ReplyDelete
  8. Hi Simon, thanks for the feedback! To expand the range of notes, one could re-implement the get_note() function. As it's currently implemented, and as you correctly observed, it chooses at random one of 5 notes defined in the array named "notes".

    Just increasing the size of the notes array and adding more octaves of the same pentatonic scale will work, but will probably produce a lot of large jumps in pitch, and that won't sound very musical. Another interesting approach might be, rather than picking a note at random, to pick a offset at random from a set of small numbers, e.g. 0-3. Once the random number has been calculated, move that many notes left or right in the array, and output that as the selected note. The result will probably sound more melodic. I will leave it as an exercise to the reader to think about what happens when you run off the lower or upper bound of the array of notes. :-)

    ReplyDelete
    Replies
    1. Thanks for your reply. I will try to expand the notes as you suggest. But I also want to develop a MIDI clarinet based Wind Controller. I plan to make the body of a clarinet from wood & fit micro-switches. Then using a matrix arrangement my Arduino Due will determine which switch is being pressed in software. I just need to adapt the code you kindly shared to cater for a clarinet style MIDI wind controller. In time I hope to incorporate the expression/velocity & maybe aftertouch.
      The reason I want to do this is that I like to compose & record my own music using Sonar & various VST software synths. I even have a few hardware sound modules, e.g Yamaha & Casio products. I have been trying to input my tracks using a standard MIDI keyboard; but I am much more fluent playing a real clarinet. So a clarinet MIDI wind controller would be amazing. I would like to be able to play rock guitar solos on a 'digital clarinet' but route the MIDI notes on/off to a guitar simulator like Slayer, which feeds into Amplitude. In effect using a different MIDI keyboard to control .VST soft-synths. Do you have any suggestions, or are there any pitfalls i should avoid?
      Many thanks, SimonB here in a rather wet & rainy Somerset in the UK.

      Delete
    2. Simon, that sounds like a very cool project. You are aware that Yamaha and Akai both have wind controllers that are very much like this, correct? If you're mostly interested in just getting something to play, then going that route would be a lot faster. Here's the late Mike Brecker playing one of the early prototypes of the Akai EWI: https://www.youtube.com/watch?v=bGvYrzemJOw

      But if you want to build something yourself, more power to you! I think the Arduino Due has plenty of capacity to handle this project. You might also consider the Teensy microcontroller. It's smaller than the Due, is compatible with the Arduino, and has USB-MIDI support built in, so you just plug it right into a computer without all the 5-pin DIN cables.

      Another challenge is going to be finding microswitches that work well in this application. Another wind instrument experimenter, Onyx Ashanti (http://onyx-ashanti.com) has had good success using force sensitive resistors from SpectraSymbol.

      Delete
  9. Thanks for your reply. I've not tried the Akai/Yamaha Controllers you mentioned, but I've read lots about them, & viewed the video you suggested.

    That would be much easier, but are they exact emulations of a real Clarinet? I thought they had a few user-selectable configurations which are more like a saxophone? I'm not sure, so any advice would be gratefully received.

    My main problem is finance - I cannot afford either of these professional products at the mo. (My elderly 1960s S.E.Labs oscilloscope is having trouble keeping up with things Arduino, so I need to buy a new one. Probably a modern 100MHz Rigol or similar).

    Also I'd love to build & develop something myself, although I know it won't be akin to the professional commercial products that you mention.

    But I digress. I take your meaning regarding microswitches. My prototype 'feels strange' using them. And my experiments are revealing switch bounce. I can address that within the Arduino Due software, or in the hardware switch circuits too I suppose. The Arduino Due has enough program memory I think, & its clock seems to run at 84MHz. I haven't yet run the usual 'take an output HIGH, no delay, take it LOW, no delay' in a loop & measured the frequency (or period) this gives, but it must be fairly quick. I expect my 18MHz oscilloscope will not be able to display such a waveform.

    Its a shame the Arduino family don't have built-in MIDI support like the Teensy. I read your earlier tutorial using these boards. So the MIDI data is sent somehow on or within the USB connection, which music creation software applications such as Garageband, Cubase, Logic or Sonar are able to receive?
    That sounds very desirable, as I wouldn't then need to build the MIDI transmitter current-loop hardware & its 5 pin DIN MIDI Out interface, as it would flow via the USB from the Teensy I guess?

    Regarding keys & playing, I will have a look at your suggestion of maybe using force sensitive resistors instead of micro-switchery. It looks like a more natural way of doing things. I was planning on using the micro-switches as an interim, to prove the design works, & then finding something more suitable.

    I originally had began using PIC MicroControllers, but became buried in 500 page data-sheets, 'Mid-Range Product Data-sheets', writing in Assembler, getting a PIC Programmer, debugging etc. But then I discovered Arduino, where I could write in C, compile & upload my 'sketch' simply using a USB cable, & let the very comprehensive Arduino hardware help me get my project developed & working.

    Still, I have lots to do; lots to think about; & some choices to make. But thanks for your help & for starting these very good tutorials. I'm going to check-out that Teensy website now.

    Thanks,

    Simon B.

    ReplyDelete
  10. >That would be much easier, but are they exact emulations of a real Clarinet? I thought they had a few user-selectable configurations which are more like a saxophone?

    I'm not a woodwind player, so I don't know if there's a clarinet fingering - I suspect it only does saxophone.

    >So the MIDI data is sent somehow on or within the USB connection, which music creation software applications such as Garageband, Cubase, Logic or Sonar are able to receive?

    That's correct. There's an official industry spec for how to send MIDI data over USB, and the Teensy implements that. I can plug my Teensy-based instruments into my Mac and they are recognized by GarageBand, Logic, and Ableton Live with no problems.

    Re: debouncing. I think you also will want to think about "de-glitching". By that, I mean that you need to think about how to ignore "intermediate states" of fingering transitions. For example, if you are moving from a C to a B (middle finger of left hand to first finger of right hand), it's quite likely that for a brief instant you either have both fingers down, or no fingers down. You don't want to trigger a note on event for that. One way would be to introduce a small delay buffer and ignore any fingering states that don't last at least milliseconds. By making configurable, you can allow the player to fine-tune this. More experienced and precise players can have be small, but beginners might want to dial in a larger .



    ReplyDelete
  11. I'm still patiently awaiting the completion of this blog series. :)
    Fantastic primer on the subject.

    ReplyDelete
  12. Thanks, Daniel, for the compliment and gentle nudge. I'll try to get back to working on this soon!

    ReplyDelete