Stop Midi Translator running as background task after crash?

It's rare, but MT sometimes still crashes, with an error message stating there has been a buffer over-run. Ableton is not affected (continues to run).  In fact MT is running in a hobbled way, with some of my controllers still active, but others dead.

Sometimes I'm able to exit and then relaunch MT, but the dead controllers are still dead in that case  Other times, nothing happens when I try to relaunch MT after exiting.  The task manager shows MT is still running as a background process, but "End Task" doesn't stop it.  I wonder if MT would start up again if I were able to use "End Task" successfully.

Rebooting my laptop does allow me to start MT again, but this is not an agreeable thing to do when playing a set.  If I were able to get MT to start again with Ableton continuing to run, that would be great.

Any advice?

I guess the first thing I would try to figure out is why you are getting buffer overruns in the first place,  Maybe you have a infinite midi loop happening here. I might be able to tell if I could see your project file.  Also, what version of MT Pro are you running and on what platform?  I’ll also pass this question along to Florian to see if he has a way to capture a crash dump for analysis.

 

Yes… I have to admit that I asked a related question about 1 month ago, and Florian suggested that I post my code. I would have done that but for two reasons:
First, I thought I had found the reason for the overruns.
Second, the code is pretty big – exporting as text yields 42,000 lines (which, of course includes comments). I couldn’t imagine anyone having the patience to look through it all.
In terms of running the code to find the problem, this, too might be complicated because I have an odd assortment of controllers all running into MT.
But, of course, I’ll gladly send you the code if you believe there’s a way to check for problems.

…also, it happens rarely, so it would take patience (or knowledge?) to track it down. However, I now think there might be a way to force it to happen, or at least to make it likely to happen within a session.

I will be happy to take a quick look at your code and look for most obvious causes. I get it that you are a power user. I hope you have a platform with enough horsepower to keep up with all of your midi messages flying around. I would not export it, just post as normal project file. I anticipate I would spend no more than 1/2 hour looking at it if you think another set of eyes would help. Did Florian advise if here was a way to set up for crash dump analysis? What version of MTP are you using and on what platform?

I think I could also force an overrun but that would not necessarily help determine if there is a problem in the code unless you can already pinpoint it to a certain sequence of events while you are running normally.

Is it possible that a timer can cause an overrun? I use a timer to block repeated sends of midi program change messages to my Novation SL61. If I don’t limit the sends this way, I get buffer overruns in the Novation. I’m now going to disable that module of the code to see if that helps, though I’m not sure I can force the overrun in the first place. I believe that the timer may be invoked several times before it times out, but I’m not sure.
I have a quite fast (3Ghz) quad core laptop. I have the latest MTP for Windows.
If there’s a way to trace a crash, that would be great, though the crashes don’t actually make MTP stop running altogether every time. Rather they generate the error message and then no longer respond to some of my controllers.

When MT stops responding to certain controls, but keeps running, can you pinpoint the problem to a specific midi port or control surface? Maybe it is the midi driver for a given port itself that is causing a problem. My understanding of timers is that if a timer it gets set again before previous one runs out, it just resets itself (starts again from the beginning). Do you have multiple translators changing the state of the same global variable. I read in the tips somewhere that there could be instances where two translators settng the same global variable could cause a lock condition so it is best to limit the number of translators that make changes to a given global variable.

Your last statement is kind of scary. I certainly do have many places where the same global variable is evaluated and changed by several translators, in fact my code depends on this kind of interaction quite often. Hopefully I haven’t caused a problem this way.
I do believe the problem could be one or maybe all of the Akai APCmini’s that I use.Disconnecting them, though, won’t help, because I can’t really run my code in any meaningful way without their input. I’ll let you know what I find as I dig into this more deeply.

OK, if you are using a global variable to toggle whether to enable or disable a given controller, this might be it. When MT Pro locks up next time, maybe do a dump of global variables to see if something there looks suspicious. That is, if you can actually do this of course. I try to ensure minimal interaction on global variables by limiting what changes them to just a few translators especially when various timers are involved and you cannot guarantee their synchronicity. Two translators could be trying to change the same global variable at the same time which can put a global it in an unknown state.

Also, do you have a translator the reinitializes all global variables? I usually include a preset for initialization as follows in my projectects
In – Program Start – Out start timer init
In – Computer Eescape key – Start timer init
In – Timer Init – Rules initialize all global variables.

With this, I can ensure my global variables are in a known state at project start and when I hit my computer ESC key . Just a thought but again hard to tell without looking at you code. If you don’t want to post it you can email me a bome@sniz.biz

Regards

Steve
Bome Q&A Moderator and independent Bome Programming specialist

Answers to your questions below… but I wonder if you think that using the MT Midi router (Through) to send Midi Clock from one instance of Live to a second instance of Live (running on the same computer) might clog up the system and cause my problem.  I’m trying to understand why Rewire isn’t handling this by itself.  Maybe Live doesn’t like to send and receive to itself via Rewire, though I’m having no problem sending data to Live via Midi, and the MT Router does transmit the midi clock successfully.

The globals are never used to enable and disable controllers.  I do send program changes to the Novation 61SL in order to change a few parameters in the keyboard – mostly, though, to change what is displayed on the 61SL screens.

I don’t initialize all globals to 0.  I believe they are automatically set to 0 on launching MT.  But I do set many of the variables to values I want them to have on startup.

I’m glad to post my code once I’ve exhausted all ideas for fixing the problem.  I can’t imagine you plowing through 40 thousand lines of code, and want to save you the trouble if I can fix this myself.

Not seeing your project, I can only speculate. I’m thinking if you are getting midi overruns, perhaps there is something misbehaving and trying to figure out what it is. Obviously there is a lot of incoming traffic that MT Pro can’t handle if MT Pro is indeed reporting buffer overruns. The incoming traffic could be from any input device, DAW, synth, etc. It could also be self generated by MT Pro if :
1) you have a midi route set and think you are swallowing certain messages but they are not getting swallowed hence generating more traffic. I recently discovered for swallow to work properly you have not only have to identify the incoming message but swallow only works if also the rules result in an outgoing message of some sort.

2) You have timers generating midi traffic that is ultimately feeding back into MT pro (on top of your normal traffic)

I assume you have a feel for what is generating midi traffic and maybe you can turn of one stream at a time until the overruns stop. The difficulty lies in the fact that overruns do not happen very often.

I have not experimented with either multiple instances of live or passing clock data from one instance to another. With that said, it would probably be best to have a single source of clock data feeding all of the downstream devices.

As far as globals, if they do not either interrupt or enable flow of messages, you are probably OK. If you are using them for status only, then the only likely side affect would be that status not displaying correctly.

Yes I think globals (but not locals) are initialized to zero (based only on observation). It is simply just good practice to initialize globals at project start in case behavior changes in the future. It also ensures all of your globals are well documented.

Although it may impact performance, it may be a good idea to turn on logging as a troublshooting step. By enabling logging it actually might both slow down MT pro enough to force and overrun and also give you opportunity to see what is happening when the overrun occurs.

I’ll stand back from troubleshooting for now until you are ready. I think Florian is travelling which is why he has not yet chimed in with his ideas.

Regards
Steve

bome@sniz.biz

Independent Bome Programming Specialist

Bome Q&A moderator

What you said here is interesting:

“1) you have a midi route set and think you are swallowing certain messages but they are not getting swallowed hence generating more traffic. I recently discovered for swallow to work properly you have not only have to identify the incoming message but swallow only works if also the rules result in an outgoing message of some sort.”

There are places where I use the “stop processing after executing this translator” option.  I always have no output in these translator, since usually, my intent is just to stop execution of a series of subsequent translators with the same input. I’m not intending to create an action.  I’ll look at this more closely.

I’ve now figured out a way to avoid using the MT router as a conduit for Midi clock messages from one instance of Live to the second instance.  There’s a new utility in Live called Link, which is meant to allow any computer with Live to sync with another if they’re on the same network or have a direct connection (maybe ethernet?).  At any rate, I’m able to sync the two instances of Live using Link.  Maybe this will stop the overruns.

I should also say that I’ve been running MT with view log always engaged since I’ve been constantly messing with the code and needing to see what’s going on.  I’m now turning off view log as well.

Thanks for your time, and I think it’s the right thing for you to not spend so much time right now, since I’m not out of ideas yet. I’ll post the code if I’m at the end of my rope, though, of course, you’re welcome it at any point.

I don’t think stop processing will stop any default routes. It only stops all translators down the line that have the same incoming trigger. Let me know when you are at the end of your rope. Meanwhile I’ll standby.

Hi fingerlight,
42k lines is quite a handful indeed! So I agree with Steve that preventing the buffer overflows will probably be a good solution. It’ll make your project work better, and it’s likely to prevent the “crash”.

But it might not be obvious how to change your project to prevent the “crash”. I assume you have looked into optimization chapter in the MT Pro user’s manual?

The crash is a separate issue, and indeed very annoying. We’ve had another report exhibiting this behavior, so there is a real issue. Of course, a buffer overflow is bad in general, but it leading to a zombie-MIDI Translator with zombie MIDI ports is much much worse. So it would be great if you could send me (via email) your project file and then the kind of data that you send to it. You can just use the Log Window and ONLY activate MIDI IN. I assume, you’re using MIDI as only input. Then I can probably recreate your scenario and blast those MIDI messages at the speed of light at MIDI Translator to (hopefully) trigger the buffer overflow.

We can also send instructions to you for creating crash dumps, which usually helps us to very quickly find the cause and fix it (or work around it if not caused by MT Pro). But in this case I believe that MT is not really crashing (i.e. process is terminated), so this method of remote debugging won’t work.

Thanks!
Florian

PS: the internal queue in MT Pro is kept intentionally small, to guarantee realtime operation. But for some users, the delay is not sooo important. So maybe, the queue size should be user configurable. I add that to the feature request list.