Animals respond to whole-field visual motion with compensatory eye and body movements in order to stabilize both their gaze and position with respect to their surroundings. In zebrafish, rotational stimuli need to be distinguished from translational stimuli to drive the optokinetic and the optomotor responses, respectively. Here, we systematically characterize the neural circuits responsible for these operations using a combination of optogenetic manipulation and in vivo calcium imaging during optic flow stimulation. By recording the activity of thousands of neurons within the area pretectalis (APT), we find four bilateral pairs of clusters that process horizontal whole-field motion and functionally classify eleven prominent neuron types with highly selective response profiles. APT neurons are prevalently direction selective, either monocularly or binocularly driven, and hierarchically organized to distinguish between rotational and translational optic flow. Our data predict a wiring diagram of a neural circuit tailored to drive behavior that compensates for self-motion.