The 17-year-old duo were left in floods of tears after being told they were being removed from the hit ITV show pending an investigation.
It is yet unclear whether the heartbroken act, from Penarth, Wales, will be allowed to rejoin next week.
Details surrounding the decision are still unclear.
But X Factor sources claim it relates to Josh, who has already hit the headlines for allegedly attacking his ex-girlfriend Amelia Clarke at a New Year's Eve party last year, which he denies.
Panic stricken X Factor bosses called a crisis meeting before deciding to suspend both of them.
All the other contestants have now been told.
The brothers had already rehearsed their version of DNCE's Cake By The Ocean this morning.
They were just about to perform their final dress rehearsal when they were told the news around 3pm this afternoon.
Both brothers have since been removed from the X Factor's £3.6 million contestant mansion and checked into a near-by hotel.
Our X Factor source said: "Everyone is in total shock and no one can quite believe it.
"New information came to light early this afternoon and as a result bosses felt they had no choice but to suspend Brooks Way until further investigation."Not only are the brothers heartbroken and in tears but all the contestants are totally devastated."
11 acts will perform tonight with one of them getting the boot in tomorrow night's show as normal.
Mentor Louis Walsh was also said to be devastated after getting close to the boyband, who he believed could win the show.
An X Factor spokesperson confirmed Brooks Way would not be appearing on tonight's show due to "Unforeseen circumstance."