This paper proposes a Gaussian-PSO algorithm which provides the optimized parameters for Feed Forward Neural Network. Recently the Feed Forward Neural Network is widely used in various applications as a result of its advantages such as learning capability, auto-organization and auto-adaptation. However the Neural Network has the disadvantage itself to slowly converge and get easily trapped in a local minima. In this paper, Gaussian distributed random variables are used in the PSO algorithm to enhance its performance and train the weights and bias in the Neural Network. In comparison with the Back Propagation Neural Network, the Gaussian PSO-Neural Network faster converges and is immuned to the local minima.