We tested whether the heritability of heart rate variability (HRV) under stress is different from rest and its dependency on ethnicity or gender. HRV indexed by root mean square of successive differences (RMSSD) and high-frequency (HF) power was measured at rest and during 3 stressors in 427 European and 308 African American twins. No ethnic or gender differences were found for any measures. There was a nonsignificant increase in heritability of RMSSD (from 0.48 to 0.58) and HF (from 0.50 to 0.58) under stress. Up to 81% and 60% of the heritabilities of RMSSD and HF under stress could be attributed to genes influencing rest levels. The heritabilities due to genes expressed under stress were 0.11 for RMSSD and 0.23 for HF. The findings suggest that, independent of ethnicity and gender, HRV regulation at rest and under stress is largely influenced by the same genes with a small but significant contribution of stress-specific genetic effects.