The Psychology Behind "Relax, it was just a joke. Can't you take a joke?"
The Psychology: This is often called Schrödinger's Joke: It's an insult when you get upset, but a 'joke' if you call him out.
Subconscious Intent: This is a test of your boundaries. He is seeing how much disrespect you will tolerate under the guise of humor.
How to Respond
Stop laughing. Look him dead in the eye and ask: 'Explain the funny part to me. I don't get it.' Making him explain the insult strips away the disguise of humor.