Collective modes and stability of Bose-Fermi mixtures with a BCS-BEC crossover

We investigate an ultracold Bose-Fermi mixture with a Feshbach resonance between two hyperfine states of fermions. Using a functional integral method, we calculate collective modes associated with fermion pairs and bosons in the superfluid phase. We derive a stability condition of the mixtures that is valid in the entire region of the BCS-BEC crossover. This stability condition, which smoothly connects the well-known results obtained in both the BCS and Bose-Einstein Condensate (BEC) limits, shows that sufficiently strong fermion-boson (FB) interactions destabilize the mixtures. In order to investigate the consequence of this instability, we study the ground-state energy of both the uniformly mixed and the phase-separated states. We find that FB repulsion induces a phase separation, whereas FB attraction should cause a collapse of the mixture. The possibilities for the experimental observation of such instabilities are also discussed.