The objective of this study is to develop an effective emotion recognition system based on ECG. The proposed emotion recognition system is capable of differentiating four kinds of emotions, namely neutral, happiness, stress, and sadness, based on the heart rate variability (HRV). Ten male subjects were involved in the study. Both visual and auditory stimuli were used to stimulate the emotions. Four categories of HRV features, namely time-domain, frequency-domain, Poincare plot, and differential features, were exploited to characterize the physiological changes during the affective stimuli. The support vector machine (SVM) was employed as the classifier. The genetic algorithm (GA) was exploited as feature selector. Without feature selector, only 52.2% recognition rate was achieved. However, with the GA feature selector, an optimal recognition rate of 90% was achieved. Compared with other user-independent systems published in the literature, the proposed method achieves an accuracy of 90% which is demonstrated to be the most effective for discriminating four kinds of emotions with user-independent design policy.