[critical] RB 4.7.3.4 broadcasting not working

[Christian]

Hi Dmitry,

Long time since I have not been upset by RB but those times are back 
I upgraded from RB 4.17.1.0 to RB 4.7.3.4
After about 1 minute of operation Winamp was running out of buffer and kept rebuffering all the time.
It apears that this version fails to stream properly to all my servers. If I stop Strereotools it's getting better but still runs out of buffer after a while.
CPU usage seems to be the same than when running RB 4.7.1.0.

I then decided to downgrade back to running version 4.7.1.0
ALL CONFIGURATIONS WHERE GONE!!! 
Only schedule tasks remain. Everything from configuration panel is gone and reset !

Christian
http://www.clubbingstation.com

 

[djsoft]

Downgrading to a previous version will reset the settings except for the scheduler list. I'm sorry for this.

Regarding the servers, please give more details on configuration:
- what soundcard are you using, or is it operating with no sound card? If the latter, you could try turning on the option "Do not limit decode rate" in the Broadcast section.
- what kind of server are you using and how many server encoders are set? One slow server could slow down the transmission to all servers.

You said that turning off StereoTool makes things better. Turning it off frees some CPU power... Maybe there's not enough CPU resources? The issue could be that computer is overloaded and it's unable to send data to servers fast enough. How much CPU usage Windows Task Manager shows?

 

[Christian]

Hi Dmitry,

No soundcard it's a virtual machine.
Inital setup is one VCPU but I change to two but still same issue. Initial CPU usage arround 70% with 1 VCPU, around 35% with 2 VCPU
I run 6 streams at the same time both Shoutcast and Iceast plus one stream to Radionomy.com servers. RB 4.7.1.0 can handle that with no problem with 1 VCPU. Did not test 4.7.2 Previous version 4.6 was also capable of that with no problem.
Did not change anything else.

Christian

 

[djsoft]

Please make sure that "Do not limit decode rate" option is turned on.

You should also lower CPU usage, 70% is too much. Open Settings->View, check the "Turn off visualization" option. Then minimize RB window to prevent redrawing, this will lower CPU usage even more.

Also... it could be a coincidence... as you're using virtual server, probably someone else on the same real machine started using more network, CPU or memory resources so uploading became slower. You can check the "Statistics" in RB, there's "Data encoded/sent" column: the numbers should be equal or at least "sent" should catch up with "encoded".

 

[Christian]

It is not a co?ncidence. I am managing the host and I did not change anything on it. I have two RB server on the host, the other RB server that remains on RB 4.7.1.0 was and is still running fine.
The host is dedicated to the Radio Station, running the 2 RB server on Win XP pro SP3, a linux server running our web site,  a second Linux running IceCast server. The shoutcast servers are on a different physical machine.
The RB Windows server are not directly connected to the Internet but can see the Icecast server through specific 2Gb vLan on the host.
Once again RB 4.6 and 4.7.1 where fine. There must be something different now with RB 4.7.3. I can try "Do not limit decode rate" but this can only be a workarround. The cause of the issue is somewhere else. Visualisation is already off. Changing "Do not limit decode rate" need a restart and I no not want to disconnect listeners.

Christian.
http://www.clubbingstation.com

 

[djsoft]

Once again RB 4.6 and 4.7.1 where fine. There must be something different now with RB 4.7.3. I can try "Do not limit decode rate" but this can only be a workarround.

There's one difference in 4.7.3 - Windows Media Server encoders were added. But handling of Icecast/Shoutcast servers was not changed. So if you're not using WMS, it operates exactly as RB 4.7.1.

So I think you should try the "Do not limit decode rate" option, most likely it will solve the problem.

 

[Christian]

Sorry Dmitry, but I am sure there is a difference. I am not using WMS but trust me since i am back to 4.7.1.0 both server are running fine without enabling "Do not limit decode rate".
It could only be a workarround if it fixes the issue. I am not willing to try as I will have to once again disconnect all listeners, upgrade RB to version 4.7.3 and if not fix downgrade again.

 

[djsoft]

I've checked the changelog for source code, and there was a change in the decode rate limiter feature. So you're right about different operation.
Now the limiter is more accurate, but on a heavily loaded computer (70%+ CPU usage can surely be considered as "heavily loaded") it can operate incorrectly sometimes...

So turning the rate limiter off (eg using the "Do not limit decode rate" option) as I suggested before should solve the issue. I'm 99% sure. And this is not a workaround - all those options exist because program operates in different environments, so you're just using the options according to your situation.

About listeners disconnect, for Shoutcast you can use sc_trans which will keep listeners always connected even if the source stream is offline. I'm not sure Icecast has such a feature...

 

[Christian]

Ok, I will try later today. But please keep in mind there is a potential issue here.
I tried affecting 2 VCPU so average loads droped to 34 - 40% that is more than acceptable and still have the exact same issue. I think it is linked to the fact I am running many encoders at the same time, but this is just a guess.

Christian

 

[djsoft]

Ok, I will try later today. But please keep in mind there is a potential issue here.
I tried affecting 2 VCPU so average loads droped to 34 - 40% that is more than acceptable and still have the exact same issue. I think it is linked to the fact I am running many encoders at the same time, but this is just a guess.

Christian

35% is average load. 70% on one core, 0% on another - average 35%. Broadcasting is done on one core... So adding another one doesn't give much. Please let me know how it functions with option turned on.

 

[Christian]

I will wait untill Sunday, the day we have the less listeners.

Christian

 

[Christian]

Hi Dmitry

    Bad news! Even with "Do not limit decode rate" enable it is exactly the same issue. All devices are at "No device" and there are no sound card anyway.
RB 4.7.3.4 faills to broadcast to my servers. Once again RB 4.7.1 is fine for the exact same configuration.

Christian

 

[djsoft]

That's really bad... I was sure that problem with rate limitation...

I added this to a bug list. We'll try to recreate an environment similar to yours and see where the problem is.

 

[Christian]

Hi Dmitry,

  Actually I am not so sure anymore that the bug apears between 4.7.1 and 4.7.3! I had to modify streaming servers configuration and for that tryed to allocate more streaming server to the same RB.
It seems that even RB 4.7.1 have the issue when increasing the number of streams above 7 in my case. Activating the "Do not limit decode rate" seem to get things even worst and not better as expected.
Current config is Win XP pro SP3 2 vCPU 512Mo dedicaded (no shared) RAM. RAM usage is arround 300Mo Both vCPU are arround 35%. I use both Icecast and Shoutcast servers
RB run 2 mp3 @ 128, one mp3 @ 320, 2 AAC+ @ 64, 2 ogg @ 64 -> OK on RB 4.7.1 NOK on RB 4.7.3
Increasing number of streams causing the issue on RB 4.7.1 as well.

Christian

 

[djsoft]

I think the issue could be the virtual machine. Can you try running it outside of virtual environment?

 

[Christian]

Hi Dmitry,

  I had some very bad time last week with one physical server crash and the second server that is running the virtualyser also have some storage issue. I am rebuilding a new virtual environment with RAID1 host server. I will also upgrade RB server from Win XP to Win7 on a fresh Windows install.
Running on physical servers would cost too much. Also virtual servers are easier to manage and to backup, ...
I should have the new infrastructure up and running by the end of the week. I will let you know if I reproduce the issue.

Christian

 

[djsoft]

Yes, physical servers cost more... But maybe try running it on a physical one for a little time, just to see if virtual environment is the problem?

 

[Christian]

Hi Dmitry,

    I did some more testing on a new virtualizer and fresh VMs still on Win XP pro SP3.
I noticed the following:
With 1 vCPU I can run up to 85 - 90% CPU then I start having problems
With 2 vCPU both are having the same loads but I start having problems after 43%.
Running Stereotool as VST plugin is a little better than as a winamp plugin.
The CPU load of the host server does not raise above 30% when testing

When running Stereotool VST I get the following error on RB exit preventing RB to close:
error; "Access violation at adress 02351c60. read of adress 02351c60.";

This is really strange I cannot go over 40% with 2 vCPU! If I open another application I can go over 60% on both vCPU but not with RB!

Christian

 

[djsoft]

An "exit bug"appears on some systems, it's already a known bug and it will be sorted out in the next update.

Regarding the CPU load, RB uses 1 primary working thread which yields about 90% CPU load. One thread can run only on one core, that's why you have one core doing all the work, while other core(s) are not loaded much.
It's not an issue on a real machines, for example on Intel Core i3 it's only loads one core for about 1-3%... About 0-1% on more powerful machines. So numbers of 40+% look large to me.