I guess this is the X-over crossover settings? (autocorrect I guess!).
Check that there’s no filter active and that the output though the main board sounds like the headphone output. There could also be an issue with the output (bad caps or the output is out of phase etc). It’s been reported in some threads that the top end isn’t as clear to some and could be clearer and I was one of the ones that noticed it too. A manufacturing defect can make an output sound bad and if it’s REALLY bad (as it sounds) then that could be a possibility.
I have a launch day Prime 4 and found the XLRs sounding so bad I had to run it via the RCA Master Out in the club. This isn’t a knock on the gear as I still have it and love it but when I took it into the club it sounded like the life was sucked out of it yet RCA output sounded great. I have a residency seven nights a week in this one place so used it time and time again. I think something it’s phasing yet the DJM-850 that I take the cables from sounds fine.
I think something somewhere is funky in my unit but I never took it back. I just connect via RCA.
Here is one thread of a few: Prime 4 - harsh/Muddy sound quality when loud? - #6 by MrWilks