SerialPIO stuttering due to XIP cache miss?

[myklemykle]

Hi, I found a weird behavior today and it's got me wondering if I understand how PIO actually works on RP2040. This is probably not a fault of the pico core but I'd appreciate any thoughts on it.

Basically I have code that sends a MIDI clock message regularly through SerialPIO, and that's always worked fine. I recently added a line of code right after the midi.sendClock() call, to call a function blinkLedsOnce(), just to blink some LEDs. But adding that function call caused MIDI to glitch out in a particular way.

A MIDI clock message is a single byte over a serial line: 0b11111000; encoded on the wire it just looks like a single positive pulse held for a duration of five bits at 31250 baud. But after I added this call to blinkLedsOnce() I started seeing that positive pulse held too long, which the UART on the other end would read as 0b11111100 or 0b11111110 or 0b11111111 . It looked as if the PIO was hanging briefly in the middle of sending a byte.

I eventually found that when I wrapped the call to blinkLedsOnce() in __not_in_flash_func(), the problem went away completely.

I don't get that. I guess that blinkLedsOnce() was being fetched from Flash into the XIP cache before I pinned it to RAM. But can a cache miss actually cause a stutter in a SerialPIO transmission in the middle of a byte? I thought that PIO had its own FIFOs and were driven directly from the system clock.

Here's the relevant code, it's pretty basic I think.

#include <MIDI.h>

extern SerialPIO SSerialRing1;
extern SerialPIO SSerialTip1;
MIDI_CREATE_INSTANCE(HardwareSerial, SSerialRing1, MIDI_OUT_R1);
MIDI_CREATE_INSTANCE(HardwareSerial, SSerialTip1, MIDI_OUT_T1);

void clockTick(){
  MIDI_OUT_R1.sendClock();
  MIDI_OUT_T1.sendClock();
}

#define LED_CLOCK_BLINK_DIV 8;
void __not_in_flash_func(doMidiClock_cb)(){
  static int clockCount = LED_CLOCK_BLINK_DIV;

  clockTick();

  if (--clockCount==0) {
    clockCount = LED_CLOCK_BLINK_DIV;
    blinkLedsOnce(BLINKLEN_US);
  }
} 


//void blinkLedsOnce(int blinkLen){
void __not_in_flash_func(blinkLedsOnce)(int blinkLen){
  if (loopTimer_us - lastPing_us > (hc.measureLen * 15 / 10))  // if the latest radio packet is late
    blinkLayer.blinkOnce( 0xff,0,0, blinkLen);  // blink red
  else
    blinkLayer.blinkOnce( 0,0xff,0, blinkLen); // blink green
}

[earlephilhower]

You can see all the source for the SerialPIO and the PIO here in the repo. The core does some computation and stuffs whole bytes into the PIO FIFO. The PIO pops out a bytes at a time to a temp register and shifts it out, free running. PIO instructions live in local RAM in the PIO and aren't affected by flash operations.

If power and clocks are not interrupted, the bytes will shift it as sent in. If that blink library messes with the PIO enable or PIO clocks or stuffs data into the wrong FIFO then it could mess up the send, but not otherwise.

Moving things to RAM may just make it do that corruption during transmit idle time, so the glitches don't affect output.

[Andy2No]

@myklemykle, Where does blinkLayer.blinkOnce() come from? I can't see an include for a library, or any other sort of reference to it in your code, just the call.

Unrelated to the question, but are you sending the MIDI signal to both the tip and ring of a 3.5mm connector? Was that a workaround for capacitive coupling, in which case is one of them inverted in hardware?

[earlephilhower]

Good point. If there is some physical issue because a capacitive interface is being faked, and you're messing with an LED which is going to need a delta current from the regulator, you could just be seeing injected noise.

[myklemykle]

Okay, I apologize once again for thinking I had this properly characterized when I didn't. While testing over this I've often made a change in code, flashed the board, saw a change in behavior, and presumed the two are connected. But now that I'm looking closer I find that it's probably not that particular code change that's making the difference. I am just seeing some very inconsistent pass/fail behavior on the PIO ports. They sometimes start to work again after a firmware change, but then they always stop working after the next power cycle.

Below is a simpler example sketch. It shows how I'm setting up the MIDI devices and SerialPIO -- maybe that's no longer the right way to do it?

The first time I flash a board (Adafruit Feather RP2040 RFM) with this sketch, I see the midi clock signal on the scope, as well as the blinking LED and the messages on the USB Serial port. But then when I cycle power on the board, the MIDI clock signal goes away -- but I still see the blinking LED and get the serial port messages.

If I change the code and recompile the sketch, MIDI clock sometimes reappears again, but only until the next power cycle. It's that behavior that fooled me into thinking I'd found the cause. Instead, I'm now wondering if there's something going wrong with the initialization of SerialPIO?

I've confirmed this weird behavior across several different boards, so I don't think it's hardware related. And in other sketches I've confirmed that PWM and SIO through those same two pins are still working; only MIDI over SerialPIO is having trouble.

// MIDI:
#include <MIDI.h>
#include <Adafruit_TinyUSB.h>

// PIO serial ports:
#define PIN_TIP 26    
#define PIN_RING 27  
SerialPIO SSerialTip1( PIN_TIP, SerialPIO::NOPIN ); // tx only
SerialPIO SSerialRing1( PIN_RING, SerialPIO::NOPIN ); // tx only

// MIDI instances over SerialPIO transport:
MIDI_CREATE_INSTANCE(HardwareSerial, SSerialRing1, MIDI_OUT_R1);
MIDI_CREATE_INSTANCE(HardwareSerial, SSerialTip1, MIDI_OUT_T1);


class PI_MIDI {
  public:
    void clockTick(){
			MIDI_OUT_R1.sendClock();
			MIDI_OUT_T1.sendClock();
		}

    void begin(){
			// Send inverted logic on ring
			SSerialRing1.setInverted(true,false);
			MIDI_OUT_R1.begin(MIDI_CHANNEL_OMNI);

			// Send normal logic on tip
			MIDI_OUT_T1.begin(MIDI_CHANNEL_OMNI);
		}
};

PI_MIDI myMidi;

void setup(){
  pinMode(LED_BUILTIN, OUTPUT);	

	myMidi.begin();

	Serial.begin(115200);
        if (!Serial)
                delay(100);


	////////////
        // setup jack
	gpio_set_function(PIN_TIP, GPIO_FUNC_PIO0);
	gpio_set_function(PIN_RING, GPIO_FUNC_PIO0);
}

void loop(){
	static bool lightOn = false;
	while (1){

		for (int i=0; i<12; i++){
			myMidi.clockTick();
			delay(100);
		}

		digitalWrite(LED_BUILTIN, lightOn);
		lightOn = !lightOn;

		Serial.println("tick tick tick");
		Serial.flush();
	}
}

[earlephilhower]

I've got other things running now, so can't run your code this instant, but a general q: why are you explicitly setting any gpio functions? Those are set by SerialPIO to the appropriate mux setting.

Also, I have no idea what's going on the MIDI lib and you're running Adafruit TinyUSB to boot. Any way of dropping those? If you're saying that this is SerialPIO related, then those are immaterial and you can manually send those serial clock framing bytes you mentioned above w/o the fluff. Then a simple logic analyzer can capture the data out on each individual pin (not the cap-coupled thing, that's separate and not related to SerialPIO) and track it...

[myklemykle]

Thanks for that. It looks to me, at the moment, that when I switch to Pico USB stack the problem goes away, i.e. MIDI comes back, and is always working even when I power cycle. I simplified the sketch even more (see below) and that is still the issue. Compiling with TinyUSB breaks MIDI -- except very occasionally I will flash the board and MIDI comes back, but only until I cycle power.

Also, if I leave out the MIDI device and just send raw MIDI clock messages (0xf8) to the SSerialTip1 at 31250 baud, that always works, even with TinyUSB. That makes it look like some conflict between the MIDI library and TinyUSB.

As I mentioned before, this combo of TinyUSB, MIDI and PIO had been working fine all year. The problem came up after a group of upgrades I recently did all at once: MacOS 12, arduino-pico 4.0.1, arduino 2.3.0, probably some library upgrades as well. I did do some testing in between those upgrades, but unfortunately I didn't test MIDI output. So it's hard to say what the change was that broke things, sorry. But the IDE tells me that I'm now building with the latest versions of all the Arduino libs.

////////////////////
// MIDI
//
#include <MIDI.h>
#ifdef USE_TINYUSB
#include <Adafruit_TinyUSB.h>
#endif

#define PIN_TIP 26    

// PIO serial ports:
SerialPIO SSerialTip1( PIN_TIP, SerialPIO::NOPIN ); // tx only

// MIDI instances over SerialPIO transport:
MIDI_CREATE_INSTANCE(HardwareSerial, SSerialTip1, MIDI_OUT_T1);


class PI_MIDI {
  public:
    void clockTick(){
			MIDI_OUT_T1.sendClock();
		}

    void begin(){
			MIDI_OUT_T1.begin(MIDI_CHANNEL_OMNI);
		}
};

PI_MIDI myMidi;

void setup(){
  pinMode(LED_BUILTIN, OUTPUT);	

	myMidi.begin();

	Serial.begin(115200);
  while (!Serial)
    delay(100);

}

void loop(){
	static bool lightOn = false;
	while (1){

		for (int i=0; i<12; i++){
			myMidi.clockTick();
			delay(100);
		}

		digitalWrite(LED_BUILTIN, lightOn);
		lightOn = !lightOn;

		Serial.println("tick tick tick");
		Serial.flush();
	}
}

a general q: why are you explicitly setting any gpio functions? Those are set by SerialPIO to the appropriate mux setting.

You're right, that's not necessary in this sketch. I pruned this down from my more complex code that lets the user select between PIO, SIO and PWM on those pins.

(FWIW the stuttering problem, where MIDI messages are happening but they're garbled, I am still trying to isolate that. I don't get that problem with this sketch.)

[myklemykle]

Looks like there's a device option as well? But it should also be completely off, i think. (I hope.)

[earlephilhower]

I think it's something simpler than that. If it was using the PIO then it would actually be driving GPIOs, not the USB port. So they wouldn't interfere, really. And it's a heavyweight thing to get running so unless you're explicitly starting it up with a ton of code I'd say it's completely off.

If there are a high # of interrupts, the software-hack of driving tip and ring may not work. There is no synchronization between SerialPIO instances, at all. Just calling SerialPIOA.write('a'); SerialPIOb.write('a'); will not guarantee that on the wire they line up. If there is an interrupt between the 2 calls then it's possible you would have a delay in the 2nd write large enough to span a bit or more of time. So it wouldn't actually look like what you want on the pins. Or if you write more than the buffer free and the 1st write has to wait for the first bit to go out before writing the last half...that would completely mess things up.

You could try adding noInterrupts() and interrupts() before and after the 2 writes to help minimize the chance of them being relatively delayed at the pins.

[myklemykle]

I get what you mean. In my "real" sketch core1 has a high-freq interrupt to process audio, but the only interrupts on core0, which sends MIDI, are from the USB subsystem & whatever else comes with arduino-pico. (It does use RadioHead library to send & receive packets, but I'm doing that with polling. So I don't expect it to be using interrupts, but maybe I'm wrong.)

On my previous project, this worked to keep the MIDI signal stable:

#define PIOSTOP pio0_hw->ctrl = 0x0
#define PIOGO   pio0_hw->ctrl = 0xf

void PI_MIDI::clockTick(){
  PIOSTOP;
  MIDI_OUT_R1.sendClock();
  MIDI_OUT_T1.sendClock();
  PIOGO;
}

However that started to fail on my current project at some point, so I commented out PIOSTOP and PIOGO and didn't look closer until now. And this code may be wrong; I don't know whether SerialPIO is using pio0 or pio1 now. Also I haven't tried to confirm that the core & the pico-sdk aren't using any other PIO resources in the background.

[Andy2No]

I take it adding a hardware inverter is out of the question? That way you could just use a single PIO or UART and invert it to get that other signal. Alternatively, maybe you could hack the PIO program to add an inverted second output.

I'm tempted to think having capacitive coupling at all (if that's still the case) is going to be a problem. Since it can't pass DC, bytes sent that have long runs of 1s or 0s are going to drift down... or in the differential drive scheme, drift closer together. Adding some simple hardware to get closer to the MIDI spec schematic, would go a long way towards making it more stable/reliable.

I'm inclined to question the resistor values given there for 3.3V operation - they seem too small, to me. You're aiming for 5-10mA through an LED built into an optocoupler, based on the MIDI IN circuit given here, which typically uses a 220R resistor, in series with the LED - which you can expect to drop something like 1.5V-1.7V:

Seeing all that at once may seem daunting but the basic idea is simple - you're driving an LED in a circuit you don't have the full details of, but have a fairly good idea of the details, and you're aiming to keep the current in that 5-10mA range, more or less. The inductors shown are all optional. If you make a test circuit with a red LED and a resistor, you should see it flickering a bit, as you send messages. If you look at the signal across the LED with an oscilloscope, even a PC soundcard based one, you should be able to tell how healthy your signal is.

You could compare driving a test LED circuit directly with what you get with your current way of doing it. My guess is that looking at the capacitatively coupled version will show that it's on the edge of not working properly, even when it seems to be working.

The only time it really matters whether your MIDI signal is driven by 5V or 3.3V is when you're also trying to power something from the MIDI signal. Some mergers and Through boxes do that and the older ones only tend to work from 5V. Since the signal is a current (it's all a current loop), the input of a synth or other device that accepts DIN plug or 3.5mm TRS MIDI, doesn't care about the voltage.

[myklemykle]

I appreciate all of your thought on this. But just to reiterate: this approach to compatibility with MIDI Type A connectors (which run the MIDI signal differentially between tip & ring) has been working just fine on my current hardware all summer, and continues to work on other hardware I built a year ago. As far as I can tell, this problem only began when I upgraded my system to a combination of arduino-pico 4.0.1, Arduino-ide 2.3.0 and MacOS 12. (It looks like the Arduino_TinyUSB and Arduino_MIDI packages used in this example sketch have been stable for years.) Before that, the signals looked perfect on the scope and were correctly received by several different MIDI hardware clients. So I still suspect it's a software issue.

[myklemykle]

I'm trying to pin down what triggers this failure, where using TinyUSB kills my MIDI output.
I'm really puzzled now.

Below is another version of the example sketch.
I broke out the MIDI_CREATE_INSTANCE macro to show what it's doing.
It instantiates a MidiTransport and a MidiInterface.
The MidiTransport is a simple wrapper around the serial port,
and I can talk to it directly, as shown in this sketch.
That works with both USB stacks.

But if I uncomment the line that instantiates a MidiInterface,
MIDI fails with the TinyUSB stack. (But not with the Pico stack.)

I am not calling any method on that MidiInterface instance but the constructor.
And looking at the code, it seems like the constructor is mostly empty.

#include <MIDI.h>
#ifdef USE_TINYUSB
#include <Adafruit_TinyUSB.h>
#endif

#define PIN_TIP 26    

SerialPIO SSerialTip1( PIN_TIP, SerialPIO::NOPIN ); 

// from midi.h:
/* #define MIDI_CREATE_INSTANCE(Type, SerialPort, Name)  \ */
/*    MIDI_NAMESPACE::SerialMIDI<Type> serial##Name(SerialPort);\ */
/*    MIDI_NAMESPACE::MidiInterface<MIDI_NAMESPACE::SerialMIDI<Type>> Name((MIDI_NAMESPACE::SerialMIDI<Type>&)serial##Name); */

// These are the two equivalent definitions:
midi::SerialMIDI<HardwareSerial> serialMIDI_OUT_T1(SSerialTip1);

// When I comment out just this line, the sketch runs correctly with TinyUSB:
midi::MidiInterface<midi::SerialMIDI<HardwareSerial>> MIDI_OUT_T1((midi::SerialMIDI<HardwareSerial>&)serialMIDI_OUT_T1);

class PI_MIDI {
  public:
    void clockTick(){

      serialMIDI_OUT_T1.beginTransmission(midi::Clock); // no-op
      serialMIDI_OUT_T1.write(midi::Clock);
      serialMIDI_OUT_T1.endTransmission(); // no-op

    }
    void begin(){
      serialMIDI_OUT_T1.begin();
    }
};

PI_MIDI myMidi;

void setup(){
  pinMode(LED_BUILTIN, OUTPUT);  

  myMidi.begin();

  Serial.begin(115200);
  while (!Serial)
    delay(100);

}

void loop(){
  static bool lightOn = false;
  while (1){

    for (int i=0; i<12; i++){
      myMidi.clockTick();
      delay(100);
    }

    digitalWrite(LED_BUILTIN, lightOn);
    lightOn = !lightOn;

    Serial.println("tick tick tick");
    Serial.flush();
  }
}

[myklemykle]

AH OKAY, it looks like this was all due to Arduino caching some faulty object files! I finally stated deleting the entire Arduino cache manually between builds, and now this missing-MIDI-signal problem seems to have gone away.

[myklemykle]

No, I spoke too soon. Sorry to be thrashing on this problem but I don't know what other variables to look for here. At the moment I'm still seeing this hard to pin down behavior: as I try various changes (for instance just upgraded core to 4.0.2) and then rebuild and reflash (in Arduino IDE, after deleting the cache) I occasionally get a proper signal on both pins after flashing the hardware, but then if I cycle power to reboot the hardware, one or both signals doesn't start up. Most of the time I don't get the ring signal.

The intermittent nature of the failure keeps making me think I've found the cause, but I've been wrong several times now. At this point I can only say for sure that re-flashing with a different firmware has a good chance of waking up the PIO until the next reboot, but the actual content of the firmware doesn't seem to be what matters. And re-flashing with the same image over and over does not have that affect. Is that crazy?

At the moment I'm seeing this problem even when I use the Pico USB stack.

At the moment I've taken MIDI completely out of it (see below), and am just writing raw clock messages to Serial. And I still see this problem.

I am deleting the entire Arduino cache & forcing a complete rebuild every time.

I have seven different instances of this Feather board. I've been swapping between them to see if it's possible that there's bad hardware, but they all behave the same.

I guess it's possible that the code I'm running is somehow frying the PIO instances so that they work intermittently or not at all, but I would be surprised to find out that's possible.

I guess my next step is to reproduce this on a vanilla Pico, where I can use a debugger.

I would love to find out that this is all due to some really dumb thing I'm forgetting. Any ideas much appreciated.

// PIO serial ports:
#define PIN_TIP 26    
#define PIN_RING 27  
SerialPIO SSerialRing1( PIN_RING, SerialPIO::NOPIN ); // tx only
SerialPIO SSerialTip1( PIN_TIP, SerialPIO::NOPIN ); // tx only 



class PI_MIDI {
  public:
    void clockTick(){
			SSerialRing1.write(0xf8);
			SSerialTip1.write(0xf8);
		}

    void begin(){
			SSerialTip1.begin(31250);
			//SSerialRing1.setInverted(true,false);
			SSerialRing1.begin(31250);
		}
};

PI_MIDI myMidi;

void setup(){
  pinMode(LED_BUILTIN, OUTPUT);	

	myMidi.begin();

	Serial.begin(115200);
	if (!Serial)
		delay(100);


}

void loop(){
	static bool lightOn = false;
	while (1){

		for (int i=0; i<12; i++){
			myMidi.clockTick();
			delay(50);
		}

		digitalWrite(LED_BUILTIN, lightOn);
		lightOn = !lightOn;

		Serial.println("tick tick tick");
		Serial.flush();
	}
}

[myklemykle]

facepalm<

I'm reluctant to even say this because I've been wrong so often, but ATM it looks like a flaky cable has been the cause of my intermittent signal loss for the last few days.

Since I replaced that, the test sketch is working in all configurations.

(I may or may not still have the original problem, a present-but-flaky signal in the real sketch ... returning to that problem now.)