This is probably false. Yamamoto's "rifle behind every blade of Grass" quote is probably a myth. Likely the Japanese simply never had the capability to invade the US to begin with.
The Japanese DID invade America - or at least they tried. That's what Pearl Harbor was all about. Their government had them so lashed into a frenzy that they didn't think they could possibly lose. It came as a real shock when we hit back so hard.