Variable (V) domains of antibodies are essential for antigen recognition by our adaptive immune system. However, some variants of the light chain V domains (VL) form pathogenic amyloid fibrils in patients. It is so far unclear which residues play a key role in governing these processes. Here, we show that the conserved residue 2 of VL domains is crucial for controlling its thermodynamic stability and fibril formation. Hydrophobic side chains at position 2 stabilize the domain, whereas charged residues destabilize and lead to amyloid fibril formation. NMR experiments identified several segments within the core of the VL domain to be affected by changes in residue 2. Furthermore, molecular dynamic simulations showed that hydrophobic side chains at position 2 remain buried in a hydrophobic pocket, and charged side chains show a high flexibility. This results in a predicted difference in the dissociation free energy of ∼10 kJ mol−1, which is in excellent agreement with our experimental values. Interestingly, this switch point is found only in VL domains of the κ family and not in VLλ or in VH domains, despite a highly similar domain architecture. Our results reveal novel insight into the architecture of variable domains and the prerequisites for formation of amyloid fibrils. This might also contribute to the rational design of stable variable antibody domains.