Variational cluster approximation study of Mott transition with strong spin-orbit coupling

Motivated by recent experiments on Sr₂IrO₄, the ground state magnetic and electronic structures are studied theoretically for a two-dimensional three-band Hubbard model with strong spin-orbit coupling. To treat spin-orbit coupling, local Coulomb interactions, and band structure effects on the same footing, the variational cluster approximation based on the self-energy functional theory is employed. It is found that for a relatively large coupling region, the ground state is an anisotropic antiferromagnetic Mott insulator of an effective local angular momentum J_eff = 1/2 with xy plane as an easy plane. This anisotropy is caused by the strong spin-orbit coupling along with the inter-orbital Hund's coupling. The momentum resolved one-particle excitations are also studied for the Mott insulating phase. It is found that the low-energy one-particle excitations consist mostly of the J _eff = 1/2 state, a direct evidence of a novel J_eff = 1/2 Mott insulator.