Well, as far as I understand, when you apply for EEA2 (and you did, I assume) you have to prove that your marriage is legitimate. When you apply for EEA4 you do not really need to prove old facts. Treaty rights for 5 years are only matters (both of you) and your residence in UK (again for 5 years). What happened to you is very simple thing. You have done something that made UKVI to put you in their black list. I do not expect you to give all the information, I’m sure you know the exact reason.
Every interview or home visit cost a lot of money to UKVI. Therefore, these things do not happen out of the blue they are driven by their budget. I really want to believe that this is just a mistake, but logically it does not end up.