Suffering from continuous stress can lead to serious psychological and even physical disorders. Objective stress assessment methods using noninvasive physiological responses such as heart rate variability (HRV) and electroencephalograms (EEG) have therefore been proposed for effective stress management. In this study, a novel wearable device that can measure electrocardiograms (ECG) and EEG simultaneously was designed to enable continuous stress monitoring in daily life. The developed system is easily worn by hanging from both ears, is lightweight (i.e., 42.5 g), and exhibits an excellent noise performance of 0.12 muVrms. Significant time and frequency features of HRV and EEG were found in two different stressors, namely the Stroop color word and mental arithmetic tests, using 14 young subjects. Stressor situations were classified using various HRV and EEG feature selections and a support vector machine technique. The five-fold cross-validation results obtained when using both EEG and HRV features showed the best performance with an accuracy of 87.5%, which demonstrated the requirement for simultaneous HRV and EEG measurements.