Considering that driving stress is a major contributor to traffic accidents, detecting drivers\' stress levels in time is helpful for ensuring driving safety. This paper attempts to investigate the ability of ultra-short-term (30-s, 1-min, 2-min, and 3-min) HRV analysis for driver stress detection under real driving circumstances. Specifically, the t-test was used to investigate whether there were significant differences in HRV features under different stress levels. Ultra-short-term HRV features were compared with the corresponding short-term (5-min) features during low-stress and high-stress phases by the Spearman rank correlation and Bland-Altman plots analysis. Furthermore, four different machine-learning classifiers, including a support vector machine (SVM), random forests (RFs), K-nearest neighbor (KNN), and Adaboost, were evaluated for stress detection. The results show that the HRV features extracted from ultra-short-term epochs were able to detect binary drivers\' stress levels accurately. In particular, although the capability of HRV features in detecting driver stress also varied between different ultra-short-term epochs, MeanNN, SDNN, NN20, and MeanHR were selected as valid surrogates of short-term features for driver stress detection across the different epochs. For drivers\' stress levels classification, the best performance was achieved with the SVM classifier, with an accuracy of 85.3% using 3-min HRV features. This study makes a contribution to building a robust and effective stress detection system using ultra-short-term HRV features under actual driving environments.