It sounds to me like you have, indeed, made a choice. Your belief system has diverged from the Christian church by your choice, has it not? Whether or not you were baptized into the Church of England shouldn't have any bearing on that choice. It may be that you have not yet shaken off the teachings of that Church. I can understand this, as I still find myself struggling against my Catholic training. But I would think that once one decides to abandon one path in favor of another, he has made a choice and severed the ties which bound him.
Out of curiosity, if you were filling out some form of paperwork which asked you to list you religion, what would you put down? I only ask because of my own experiences. It took me a long time before I felt comfortable writing "none" instead of "Catholic". And I could certainly understand if you were reluctant to put down "Pagan".
And please understand that I'm not trying to ridicule or belittle you. I'm truly stunned by the idea that you would consider yourself to have no choice in your religion simply because of an accident of birth.