Inducible switching between phenotypes is a common strategy of bacteria to adapt to fluctuating environments. Here, we analyze the switching kinetics of a paradigmatic inducible system, the arabinose utilization system in E. coli. Using time-lapse fluorescence microscopy of microcolonies in a microfluidic chamber, which permits sudden up- and down-shifts in the inducer arabinose, we characterize the single-cell gene expression dynamics of the araBAD operon responsible for arabinose degradation. While there is significant, inducer-dependent cell-to-cell variation in the timing of the on-switching, the off-switching triggered by sudden removal of arabinose is homogeneous and rapid. We find that rapid off-switching does not depend on internal arabinose degradation. Because the system is regulated via the internal arabinose level sensed by AraC, internal arabinose must be rapidly depleted by leakage or export from the cell, or by degradation via a non-canonical pathway. We explored whether the poorly characterized membrane protein AraJ, which is part of the arabinose regulon and has been annotated as a possible arabinose efflux protein, is responsible for rapid depletion. However, we find that AraJ is not essential for rapid switching to the off-state. We develop a mathematical model for the arabinose system, which quantitatively describes both the heterogeneous on-switching and the homogeneous off-switching. The model also predicts that mutations which disrupt the positive feedback of internal arabinose on the production of arabinose uptake proteins change the heterogeneous on-switching behavior into a homogeneous, graded response. We construct such a mutant and confirm the graded response experimentally. Taken together, our results indicate that the physiological switching behavior of this sugar utilization system is asymmetric, such that off-switching is always rapid and homogeneous, while on-switching is slow and heterogeneously timed at sub-saturating inducer levels.