Most commercial optical sorting systems are designed to achieve high throughput, so they use a naive low-latency image processing for object identification. These naive low-latency algorithms have difficulty in accurately identifying objects with various shapes, textures, sizes, and colors, so the purity of sorted objects is degraded. Current deep learning technology enables robust image detection and classification, but its inference latency requires several milliseconds; thus, deep learning cannot be directly applied to such real-time high throughput applications. We therefore developed a super-high purity seed sorting system that uses a lowlatency image-recognition based on a deep neural network and removes the seeds of noxious weeds from mixed seed product at high throughput with accuracy. The proposed system partitions the detection task into localization and classification, and applies batch inference only once strategy; it achieved 500-fps throughput image-recognition including detection and tracking. Based on the classified and tracked results, air ejectors expel the unwanted seeds. This proposed system eliminates almost the whole weeds with small loss of desired seeds, and is superior to current commercial optical sorting systems.