Yes, that is correct. It is very advisable to make sure that you always have a valid - EEA Family Permit, or - Residence Card, or - Permanent Residence Card You will not lose your rights if you have none of these, but it will be difficult to exercise your rights, like re-entering the UK coming back...