We propose a Quadratic Unconstrained Binary Optimization (QUBO) matrix distorting method to solve combinatorial optimization problems, which are very often seen in consumer applications, with high speed and high accuracy. Combinatorial optimization problems are formulated as the problems of finding the ground state of the energy function determined by the QUBO matrix. The proposed method consists of the local optimization and the distortion of the energy function by adding a constant to each QUBO matrix element with a certain probability. The probability is initially large, decreased during the method linearly, and zero at the end. The distortion process aims to make a local optimal solution to be no longer a local optimum and avoid staying in the local optimal solution with a probability. We apply the proposed method to the graph partitioning problem as a typical combinatorial optimization problem with many local optimal solutions and large energy barriers among them. We verify the effectiveness of the proposed method against the iterative improvement method and the simulated annealing.