They do state that for PR a person must have been exercising Treaty Rights for five continuous years. However, case law also states that decisions to refusal residence documents must be proportionate.
Thus, they do not state x amount of time not exercising treaty rights means you fail to meet the criteria, as each case must be judged on it's individual merits.
If she studied even two days a week and also did a language course, then that's better than not offering any proof of exercising treaty rights during that period of time.
During the times she was a student, did she have comprehensive sickness insurance?
Read below, about CSI. What I wonder is that, will UKBA then accept your sister re-registered as a student so she does not require proof of CSI?
It seems she did reapply for a document confirming residence as a student, so that's in her favour.
http://www.ukba.homeoffice.gov.uk/sitec ... iew=Binary
"EEA nationals who have already been issued with a registration
certificate on another basis (for example as a worker) and who then
became a student (but who did not reapply for a document confirming
this before 20th June) will not be treated under these transitional
arrangements. Such applicants must show evidence of comprehensive
sickness insurance for any time spent in the UK as a student. "
As she was registered as self-employed for some of the time, hopefully she can also provide evidence that she was genuinely earning money as self-employed. UKBA have demanded proof of accounts and National Insurance payments from others who have applied as self-employed.
Did she claim any benefits when she was not registered as a worker?
All your sister can do is try and present as much evidence as possible that she has exercised treaty rights for five continuous years and then hope for the best. If the application is rejected by UKBA , she could then appeal.