I suppose the problem is incorrect mixer wiring. In the result, the sound loops: music goes through MIC input and goes to mixer, then again to MIC and so on, causing feedback loop that sounds horrible.
Try without the mixer. Connect MIC to sound card input and select the input in RadioBOSS. Check how it works. If it works, add mixer to the setup.
To get rid of latency, either upgrade to Windows Vista or newer (7, 10) - it has WASAPI driver that allows low latency. Alternatively, if your card supports ASIO, use this mode; if it doesn't you can try Asio4All: http://www.asio4all.com/
Another important note. If you have hardware mixer board, probably you shouldn't be using the MIC feature at all. This feature exists for cases when there's no mixer.
In case of using a mixer I suggest a setup like this:
1) RadioBOSS playback is routed to one of the mixer's inputs.
2) You also have one or more mics connected to the mixer as well.
3) Mixer output is connected to sound card input
4) Select this input for Input 1 or Input 2 in RadioBOSS (Settings->Inputs); select this input as audio source is Broadcasting tab instead of Audio Mix (the setting is located at the top right corner).
This way, RadioBOSS will broadcast your mixer output. Please let me know if you need more information on it.