Yes, you may travel without a visa.
I´m just adding something, which should reveal how messed-up the Schengen-System is:
According to the Norwegians, you may enter:
Norwegian Embassy wrote:- You do not need to be accompanied by your EU/EEA spouse or partner.
- It is advisable to print this page and to present it together with your passport, documentation from UK Border Agency regarding your Residence Documentation (residence permit) and proof of your relationship with the EU/EEA national (e.g marriage certificate or birth certificate) at the point of departure and at Norwegian Border Control on entry into Norway in case of problems.
- If you are travelling to another Schengen country other than Norway, you should check with that country's embassy in the UK whether they also accept you as being exempt from the visa requirement.
Compare this to general Schengen-Info from a randomly picked German embassy:
German Embassy (Bratislava) wrote:- Harmonisierte Visumpolitiken der Mitgliedstaaten (gemeinsame Liste der Drittstaaten, deren Staatsangehörige visumpflichtig sind).
- Außengrenzkontrollen nach einheitlichem Standard.
Translation by me:
- Harmonised visa-policies of memberstates (common list of third-country states, whose citizens are visa required).
- Exterior-checks according to a uniform standard.
Problems:
- Schengen-States other than Norway insist on the EU-spouse accompanying the non-EU-partner
- If the rules were clear and known to everyone, why should it be necessary to "print this page"?
- Schengen aims to harmonise visa-rules. If they were truly harmonised, why should you check with any other Schengen-country? It should be the same everywhere...