He didnt invade anyone at the start, he just didnt role over and surender federal lands. The South did the invading to begin with by attacking Fort Sumter.

The precedent for Federal residing over state was decided upon long before Lincoln got into power.