Thanks to the EU/EEA regulations that came into force on 30.04.06 you have a right to live and work in the UK, assuming your EU/EEA Family Member is exercising their Treaty Rights in the UK. Indeed there is no compulsion for you to apply for a 5-year Residence Card, but if you don't, how do you prove that you have those rights, including the right to live and work in the UK? Therein lies the problem!
The preventing illegal workers guidance is quite specific about what documentation an employer, or prospective employer, should seek. And given that such employers cannot be expected to have a detailed working knowledge of EU/EEA regulations, this leaves you with a real problem.
As you say if you apply for a Residence Card you will get a letter back confirming your ability to work, but that does not help you at this instant in time. The best I suggest at this moment in time is for you to go to this BIA webpage and then download Chapter 2. At 1.1 it states :-
In other words the issue of a Residence Card to you gives you no more rights, but it does confirm that you have those rights. Suggest you print out at least part of the document, and take that to the agency, and hopefully you will be allowed to work.As confirmation of this right of residence, non-EEA national family members may apply for a residence card, which if issued, will be valid for 5 years, or shall be valid for the envisaged period of residence in the UK of the EEA national (if this is a shorter period of time). This refers to Regulation 17(6)(a) and (b) of the 2006 Regulations.
Final point .... in what way is your EU/EEA Family Member exercising their Treaty Rights in the UK?