Bomebox loses it when handling MT project

Hello,

I have an issue problem of which I have no clue what's going on. Normally I'd troubleshoot myself but I have a gig next Saturday and I would love to be have this problem resolved beforehand.

I have recently rewritten my MT project that handles my MIDI setup during live gigs. It basically handles all patch changes, volume controllers, note sends and LED feedback between my two keyboards and MIDI control surface.

I always develop with MTpro running on a laptop. All MIDI traffic goes through an integrated USB interface of the control surface except the MIDI out of keyboards #2 which comes in on the Bomebox DIN All port are individually accesible.

All works exactly as should when the Bomebox is in router mode and my laptop handles the MT project. However shit hits the fan when transferring the project to the Bomebox. MASSIVE latency, hung notes etc. And most peculiar - the web applications acts all weird. Its super unresponsive and everything takes minutes to load. Without the project its instant. Last night at band practice it appeared to by itself unload my project and forget all assigned aliases. It also acts funny on power supply and I sometimes have problems getting it to turn on.

Afterwards I switched to the setup with MTpro running on my laptop and all worked fine again. Including the BB.

What is causing this extreme behaviour - where should I start looking?

Thanks -

Hi, are you running WiFi? This is a long shot but see if you can turn it off to see if the latency clears. Maybe if there is a lot of WiFi noise the BomeBox is trying to handle the extranous WiFi traffic that is not meant for it. Also, I occasionally get something similar but usually it clears up when I power cycle my BomeBox and one of USB keyboards that draws power from the BomeBox at the same time.

Just a few suggestions, but I’ve also alerted Florian in case he has some other ideas.

 

Thanks for the help. The Wi-Fi is indeed turned on although by default I am not using it for anything but changing the settings of the bomebox. I do use it for MIDI data when running MT in the laptop and then it works well. That route I automatically removed when I load my MT project on the bomebox.

We practice in the middle of nowhere – it could be my band members smart phones trying to connect but then I would definitely expect similar problems when the Bomebox is functioning only as a router. But that’s clearly not the case.

 

I’ll try this out ASAP.

Hi, first of all, apologies for this experience! We have never heard of such symptoms, so for now I can only guess what’s happening.

In general, depending on the type of your project, it is possible (though unlikely) that it causes slightly higher CPU load, which then (temporarily) overloads the power supply. Unsuitable power supplies are the #1 cause for BomeBox problems. Note that WiFi also draws a bit power when active. As a rule of thumb, the more amps a micro USB power supply has, the more stable it will provide power to the BomeBox. The most stable power supply (also in mechanical terms) is using a Power-over-Ethernet switch or injector. The latter are available for little more than $20.

Still, flaky power does not seem to explain your symptoms well. It rather seems like CPU overload. Apart from a bug in the firmware (which I wouldn’t assume at this point), I can only think of some kind of MIDI feedback or infinite loop in your MT Pro project. Could you please attach it in this forum and list which devices are connected to the BomeBox? Hopefully we can reproduce it in our office.

Last, but not least, I assume you’re using the latest firmware?
https://www.bomeloft.com/products/bomebox#downloads

I hope we’ll solve this asap!
Florian

Thanks for the feedback – I’ve just ordered a 48V PoE Injector – if only for the solid feel of connection vs micro-USB (and plugging in a UTP cable during live setup looks cool).

I think I have the latest firmware – at least something 1.3.x – the one that supports QWERTY input.

Currently my gear is bagged – going to setup this evening, download that specific MT project from my Bomebox and I’ll share it here.

MIDI Setup:

1. Bomebox hosts MT Project.
Nord Lead 2x OUT -> Bomebox DIN IN
Nord Stage EX OUT -> BCF2000 DIN IN
BCF2000 USB Bomebox USB
BCF2000 DIN OUT #1 -> Nord Lead 2x IN
BCF2000 DIN OUT #2 -> Nord Stage 2x IN

2. Laptop hosts MT Project.
Exactly the same except:
BCF2000 USB Laptop USB
Nord Lead 2x OUT -> Bomebox DIN IN -> WiFi -> Laptop

Hi, I’ve attached a few diagrams to see if I got it right. Please confirm.

How do you have your MIDI routing (both within BomeBox and within your MT Project file). The reason I ask is if you are not careful in this configuration, there might be possibility of some MIDI loops.

On you Nord Products (which I think can support sending and receiving through multiple MIDI channels, what channels to you have configured?

And sorry, I think you said your Nord State EX is Stage not Stage 2

Finally I think you said Scenario 2 is working but note Scenario 1?

In Scenario 2, are you running project files on both BomeBox and Laptop, just Laptop, or just BomeBox? On the device not running a project file, how are your MIDI routes set up?

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

 

Another thing that looks wrong is having MIDI going into BomeBox but not going out to anywhere. Maybe I missed something?

Diagram looks right – only that BCF2000 Bomebox is bidirectional. (Which you mentioned in the second comment). I can access all the BCF’s outputs its own input though MT. Its works sort of like a MIDI interface with 3 ins and 2 outs.

No routing in MT project in both scenarios. Bomebox only has the WiFi routing (extends 2x to laptop). and only in scenario 2 (Bomebox hosts project). Everything is routed by translators.

The MT project is only running on the laptop in scenario 2 and on BB in scenario 1.

It’s indeed scenario 1 that behaves weird.

I’m going to setup my gear now and upload that project

And here the MT Project I downloaded from my Bomebox (it’s supposed to be identical to the on my laptop).

I forgot the MIDI channels :
NSeX (a classic one indeed): 13 to 16 OUT, 5 to 10 IN
NL2x 1 to 4 IN and OUT
BCF2000 16 IN/OUT (except Faders which send on the Nord’s instrument channels)

But everything is sorted by the input ports of my translators – and no MIDI goes direct through without passing a translator.

OK, once you send the file, could you do a screenshot of BomeBox routing after starting your project file there (for scenario 1)?
So as it looks, you never send performance (keyboard data from your Stage to your Lead) but you can do the opposite if using translators (Lead to Stage)

BCF is actually just used as aux control surface. If I remember it will even allow motorized fader control from your keyboards if you desire and perhaps LED updates.

Bomebox routing is empty. Actually I just uploaded a version of my MT project with the WiFi port disabled. I also disabled BUT Virtual MIDI port ( it isn’t used anywhere – i use it seperately for my DAW). Seems to be working for now. Can these ports cause the aforementioned behaviour? About the project – the BCF sends midi cc that Bome translates into patch changes for the keyboards. It also acts as a MIDI mixer for my patches volumes and I use it to assign master slave configurations. All of this is processed by my project which then returns corresponding LED signals to keep me updated on what is controlling what 🙂

Well at first glance things look OK but it might take a while to figure this project out completely. I guess the first thing I would ask is if the BCF2000 DIN #2 port is a MIDI thru port? If so, everything toing in DIN from your NS will loop right back to it and although you might not be doing anything with the MIDI coming in on that channel, it may slow things down. I would have the same question on the NS, whether it is configured as MIDI thru? If they both are then you might have a MIDI loop going on here, but I suspect not if it works OK in scenario 2. I assume when you have the project running on the BomeBox, you don’t see any routes since all routing is being handled by translators, right?

Do you have the BomeBox powered by the same power source in both Scenarios 1 and 2?

I’ll continue to poke at it to see if I can uncover any anomalies.

Just an observation, I noticed you are explicitly call out inputs and outputs at the translator level which is fine, however may be more difficult to manage over the long run. I usually manage my devices by preset, that way I can set the defaults under the preset for the predominant devices there and then if there are 1-2 translators under that preset that need to be different, I can either put them under a different preset or override them at the translator level. For me if I have 310 translators (as you do), it is easier to manage changing default ports for 23 presets than for 310 translators.

Nice project! If you are using same power source for both scenarios, I would suspect either a MIDI loop between the BCF and NS or a misbehaving USB connection from the BCF.

I have a Alesis Q88 with USB in my config and once in a while I suspect it causes me similar issues as you are reporting. I have to clear by recycling power on BomeBox after cutting power to the Q88 and then re-powering the Q88. It has only happened to me a few times but it is really frustrating when it happens.

If you are routing BCF commands through the project file to control NSEX, maybe next time it happens, disconnect one of the MIDI DIN port from the NS to the BCF to see if the problem clears. Obviously you will not be able to update faders and LED from the NS2 but you should till be able to get PCs to both of your boards.

Also, I assume the BCF is exposing the NS and NL as separate devices through the USB interface (looking at your project file). If not, I would be quite confused.

Thanks Steve – really appreciate the input. I have to be sleep/work tomorrow. I’ll check this again tomorrow.

I’m really starting to suspect that accidentally leaving that virtual MIDI port open might have caused this.

Problem not resolved. I think it occurs when something connects through ethernet or WiFi. After unloading and reloading it works ok for a while. I’m not sure what triggers it exactly but I think it has to do with my laptop connecting with it.

I just updated firmware to 1.3.1 from 1.3.0.

I’ve included an image of BCF2000 routing – there are no through ports in this setting.

The bomebox is always powered by the same micro-USB. Going to try PoE tonight though.

I also included an image of  an error I see in the Bomebox log after the problem starts. It’s included in the other image I attached.

Ignore the port not found messages since my BCF was plugged into my laptop at that point.

The UDP_MIDI errors look like a network errors. Did your project file have any ports configured as aliases to Bome MIDI network? Otherwise I assume you were running the configuration more like Scenario 2 which you said was working OK.

Perhaps at one point – at least not now. I do see my laptop but it says ‘not used anywhere in project’. Problem still persists – always a couple minutes after starting the project. – PoE works great but does not resolve the problem – I’ve disconnected my laptop from the Bomebox – problem still persists – Fixed a misassigned port (2x to BCF2000[2]) but it did not fix the problem.

Concerning the aliases – I do have that alias “NL DIN OUT” assigned to Bome Network on my laptop project. But on the Bomebox that alias is properly assigned to Bomebox DIN.

Hi, perhaps a good troubleshooting approach would be to disable any translators the produce output and then re-enable them one at a time until the problem shows up again. I’ll see if the developers have any other ideas.

I just thought of something else. I remember when I first got my BomeBox, that I had a USB device that was actually feeding back power into the BomeBox through the USB connector and causing it to misbehave. I’m wondering if maybe this is what is happening here. If you look at the Scenario 1 diagram that is not working, the really only change I see is that you have the BCF2000 USB directly into the BomeBox. Now since the BCF2000 is a device, it should not be providing power but maybe like my device (which was actually a powered USB hub), maybe it is. I think the BomeBox has protection from this but was warned that this could pose other side effects.

Maybe as a test, you can just set up your PC as a MIDI router and route the BCF2000 USB signal through the PC (project file would just be a line between input and output). Run the project on the BomeBox that way and see if the problem disappears. Bascially you would be using the PC to try and isolate a USB power feedback issue from the BCF2000.

If this fixes it, then we would have to think on how you could set up your USB connection to work from the BCF2000 directly to the BomeBox. I’m not sure if anyone makes “USB power Isolators”.

I started disabling preset one by one – narrowed it down to one preset. It appears the bug starts when ‘stop/start all controllers’ enables ‘Flash LED BCF’.

Hope something comes out of it.

Well that sounds promising. I know the APC Mini will crash if you send LED data to fast while moving the faders. The Behringer DDM4000 cannot handle incoming MIDI traffic for it’s LED’s if sending too fast (some LED lights don’t light but nothing else really bad happens). In both cases, we had to use MT Pro to slow down the MIDI data going into the controller to fix the problem.

However, if it is USB power backflow a USB islotator might do the trick. Looks like Amazon sells them (no surprise here as the sell just about anything you can think of).

I’ve just tested this with the BCF’s USB MIDI port not connected (as in physically unplugged). The problem still occurs- I notice it by the web app becoming super laggy after 2-3 minutes. I think it’s something with the Bomebox and that timer present.

What timer? Flash LED BCF?

Hmm, I was looking at your Flash LED BCF preset and I see you have a timer triggering itself which is usually not a good idea. The way are usually set up flash timers is this.

1) One shot timer to trigger a flash timer which is continuous at usually 100-500 ms.
2) Flash timer triggers an “Update LED” timer at a count of the number of LED’s you want to update.
3) LED update counter which iterates through all the leds (using count) to update them. I usually but a repeat delay of maybe 10ms so that I don’t overload data to the controller, but if you want to change the delay, you can change it in one place and you are done.

The update LED timer usually looks something like this. Say that gc is your counter, pp is the led number you want to assign for that iteration and qq is the value
gc=gc-1
// Set LED
if gc==10 then pp=15
// Set color
if gc==10 then qq=12
// Skip over rest
if gc==10 then goto “Done”

// Set Next LED
if gc==11 then pp=15
// Set color
if gc==11 then qq=12
// Skip over rest
if gc==10 then goto “Done”
… continue on until last iteration

Label “Done”

Outgoing action Raw MIDI 90 pp qq

Setting up the rules is a bit of a pain but once done, it works pretty good. Lots of copy and paste.

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

 

I mean preset ‘Flash LED BCF’- I still don’t know what about it causes the behaviour. But it appears that it starts when this present is enabled.

I’m not sure I understand correctly how you would set this up.

1 = initiate flash LED?
2 = timer interval?
3 = update LED which iterates through the LEDs with a timer with a really small time value?

My current setup was designed that flash LED was completely isolated from the rest of the project. So I can easily turn it off through ‘stop all controllers’ and quickly change the value of the repeat delay to control the amount of processing.

Works good on the laptop but seems to cause some unexpected behaviour on the Bomebox.

I’ll try to design something new for this.

Well I solved the problem by eliminating the self triggering timer. I’ve written a simplified flash LED script for tonight’s performance. I’ll restore it later.

I’m still very curious why this technique works on my laptop but makes the Bomebox behave this way.

Hi,

I’m glad you solved it. Timers can be very tricky things and if not careful, can cause performance issues. It can really depend on the host platform (BomeBox vs Laptop) and what else is going on.

I’ve written a demo (attached) that shows how I generally do my buttons.

I have two rows of 2 buttons each in this demo.

The top row handles solid on/off states (no flash timer). The bottom row uses a flash timer.

In both cases, I use a bitmap to control button states. I do this for two reasons:

1. Less global variables
2. Easier to do mass manipulation of bits with bit shifting, techniques (once you understand bit manipulation)

For the flash activity there are 5 translators

1. Start the Flash timer at 500ms  (1/2 second) intervals
2. Kill the flash timer when there are no bits set
3. The flash timer itself
4. Iterate through all of the bits setting the LED state and updating the LEDs (this is a fast timer)
5. Set Default state of LED’s when flash timer is killed (don’t leave them where they happen to be when you kill the flash timer.

I start and stop the timer using the same incoming note trigger, however the logic is to look at the LED states and only kill it when all LED’s are off (bit state is zero).  I only start the timer if it is not already running. No need to restart it.

I set a kill flag when killing the flash timer and then use this to do a final note iteration to set the default LED state after the flash timer is killed.

I know it might be a bit overwhelming, at first but I find it a much cleaner way of doing things (once you get past the learning curve).  With bit mapping technique, I can control 32 bits with a single global variable and update them all with one LED update translator timer (with lots of rules).

I hope you find this helpful!

Steve Caldwell
Bome Q and A Moderator and
Independent Bome Consultant/Specialist
bome@sniz.biz

 

Hey thanks Steve, that sounds very promising. Could be very helpful. I’ll check it out soon. Bome is beating heart of my setup so there’s incentive enough to get it maximum efficiency.

Hi this might be a cleaner solutions. Uses presets instead of variables to start and stop the flash timer. Still iterates though notes to actually flash the LED.s

 

Hi, happy to read about your solution. To shed some light what has probably been happening:
1) every time you start a timer, it creates a “thread” internally. You can ignore what that is, but it will block approx. 0.5MB RAM memory.
2) when you retrigger an existing timer, it might be that it’ll start a new timer and lets the old one die on its own (asynchronously). So for a brief moment, you’ll have 2 active “threads”.
3) now if you trigger new timers fast enough, there’ll be more and more active threads around. The BomeBox has approx. 32MB free RAM for the translation engine project, so if you create too many timers, you’ll hit a memory limit eventually.
4) A computer has much more memory available (so will not soon reach a memory limit), and its processor is much faster, allowing the processor to finish old threads faster, so that you’ll have fewer concurrent threads around with the same project.

Having explained that, it would still be nice if the BomeBox behaved better in such cases. The Bome MIDI engine should either reuse timer threads, or allow multiple timers per thread (which might hurt realtime performance). It’s now on our bug list to be fixed/improved.

Thanks for keeping calm… (I hope)!
Best regards,
Florian

Thanks for the detailed explanation, Florian! This all makes sense. It also tells me that having less timers and controlling them as I did in my project files should help alleviate the issues.

It the solutions I posted, each only has at most 3 active timers.
1) One shot timer to start the flash timer
2) Continuous flash timer to trigger the LED timer at intervals as specified in the one shot timer.
3) Counter based LED timer to actually update the LED’s. This one runs much faster and the number of iterations (LED’s to control) are determined by the variable set in the flash timer.

The flash interval is controlled by the flash timer and the LED update speed is controlled by the LED timer. No need to have different timers for each LED under control. The other positive thing about this strategy is the if you use a single LED timer controlled by the flash timer, then the LED’s flash in sync (or pretty close).

Most of the heavy lifting (rules) such as note, numbers, and velocities (colors) are handled within the flash timer, so if you need to change which LED’s to flash and what color, it can all be handled there in rules of this translator.

Steve

Thanks for the explanation – I do see how that couls result in performance issues. I’m also happy to hear that it’s on the bug list.

I’ve created a simple project with just the self repeating timer (at a 2000 ms interval so should barely lead to any processing problems) and a single note on/off translator. Same problem occurs after a short while. If i am correct this should lead to at most 1 MB RAM (old timer and new timer) in use which technically could never be a problem for 32MB memory.

I’m not sure if I recall this correct but I remember that the problem only occurred on the DIN port but I’m not sure whether that right.

I you want to post the file, I’ll have a look. Does the problem only occur if the project is running in BomeBox or does it also occur on your computer?

I don’t know how to respond with an attachment – it gives me no option to include an attachment when I try to post a comment under the appropriate response so I’ll do it like this.

I’ve included a very simple file with the self-repeating timer. I did not test this specific one my laptop but it does bug my Bomebox after about 2-3 minutes. I added a 60 second interval before the timer starts.

@Steve I downloaded your MT files but I have to admit I have no clue how bitmaps work. Do you utilize them sort of like an array of on/off values?

Thanks, I’ll have a look.

Yes, I use bitmaps as a bit array. Each bit represents a LED state 1=On and 0=Off.
Since Bome variables are 32 bit signed integers, I can control 64 pads with only 2 global variables to depict the on/off state of a given LED.

For controlling lights I typically just update a given bit’s state instead along with sending the MIDI message to the device I’m controlling. I can then use a single timer to read the bitmap and update all of the led’s by iterating through the bitmap. The only time I have to change variables is when when I change from the first group of 32 to the second group.

It is a bit of a pain to set up but once you get the logic, it is a much more maintainable project.

Steve

I think I get how it works … in theory.

Now do something practical with it. I do eat through variables quickly to store stuff so this could really improve my project.