In this study, we present an autonomous driving sys-tem that utilizes a binarized autoencoder implemented on a Field Programmable Gate Array (FPGA). The binarized autoencoder compresses the image into optimal features in this system. The recurrent neural network then determines the following control based on the feature values extracted from the autoencoder and the rotation speed of the motor. We reduced the model size by binarizing the autoencoder because of the limited on-chip memory of the FPGA. We implemented the system on an Ultra96-V2, a board with a programmable logic and processing system. The robot employing our implemented system exhibits robust control by recognizing the entire road marking and road edge line as a feature and drives autonomously along the specified route.