You don't have to deal with anything you don't feel is right. By your post I get the impression you feel he was wrong to punish you for.
If you were just kidding, truly just kidding, surely he knows you well enough to know this and therefore should not punish you for it.
Why did he get so upset about it?