The ATP-dependent degradation of polyubiquitylated proteins by the 26S proteasome is essential for the maintenance of proteome stability and the regulation of a plethora of cellular processes. Degradation of substrates is preceded by the removal of polyubiquitin moieties through the isopeptidase activity of the subunit Rpn11. Here we describe three crystal structures of the heterodimer of the Mpr1–Pad1–N-terminal domains of Rpn8 and Rpn11, crystallized as a fusion protein in complex with a nanobody. This fusion protein exhibits modest deubiquitylation activity toward a model substrate. Full activation requires incorporation of Rpn11 into the 26S proteasome and is dependent on ATP hydrolysis, suggesting that substrate processing and polyubiquitin removal are coupled. Based on our structures, we propose that premature activation is prevented by the combined effects of low intrinsic ubiquitin affinity, an insertion segment acting as a physical barrier across the substrate access channel, and a conformationally unstable catalytic loop in Rpn11. The docking of the structure into the proteasome EM density revealed contacts of Rpn11 with ATPase subunits, which likely stabilize the active conformation and boost the affinity for the proximal ubiquitin moiety. The narrow space around the Rpn11 active site at the entrance to the ATPase ring pore is likely to prevent erroneous deubiquitylation of folded proteins.