Anxiety is known to dysregulate the salience, default mode, and central executive networks of the human brain, yet this phenomenon has not been fully explored across the STEM learning experience, where anxiety can impact negatively academic performance. Here, we evaluated anxiety and large-scale brain connectivity in 101 undergraduate physics students. We found sex differences in STEM-related but not clinical anxiety, with longitudinal increases in science anxiety observed for both female and male students. Sex-specific impacts of STEM anxiety on brain connectivity emerged, with male students exhibiting distinct inter-network connectivity for STEM and clinical anxiety and female students demonstrating no significant within-sex correlations. Anxiety was negatively correlated with academic performance in sex-specific ways at both pre- and post-instruction. Moreover, math anxiety in male students mediated the relation between default mode-salience connectivity and course grade. Together, these results reveal complex sex differences in the neural mechanisms driving how anxiety impacts STEM learning.