The Fugitive Slave Law and slavery itself are apples and oranges though. The *intra* state regulation of slavery was, indeed, Constitutionally illegal. However, a state was obliged to return property to its rightful owner (I know...I know...). The banning of the importation of new slaves was Constitutional and illegal in the United States -- interestingly, it was also illegal under the Confederate Constitution. In any case, two different concepts with two different legal arguments at play. Slavery was both a Federal issue and a state issue depending upon what aspect of slavery you're looking at.
However, it is an interesting point that the Federal government did not have the power to ban slavery. I completely agree, and it's why the secessionist hotheads should have been brought to heel. It seems to surprise people, but I think secession at that point was a terrible idea. They should have waited to see what came of the Lincoln Administration before leaving the Union.