It all comes down to _why_ you are performing this movement in the first place. Because most people will be able to move more weight while wearing Olympic shoes in the front squat, and we can't argue with moving more weight meaning you're stronger, that's an argument to wear the Oly shoes - but again, it all comes down to your purpose, your goals, and the place this particular movement has within the overall framework of your training.
Plenty of people front squat in bare feet, wrestling shoes, or All Stars. If your goal is to improve your Olympic lifting performance, then I'd advise Olympic lifting shoes for you, otherwise, stick with your Vibrams.
The short answer would have been, "It depends."
-S-