Advances in low-cost portable electrocardiogram (ECG) devices have opened doors for numerous new applications, including fitness tracking, remote health, and peak athletic performance monitoring, to name a few. Many such devices, however, have been shown to be highly contaminated by movement and/or muscle contraction artifacts, which, in turn, can lead to erroneous heart rate and heart rate variability (HRV) analyses. Here, we propose a new denoising method based on adaptive spectro-temporal filtering for ECG enhancement. The algorithm relies on the so-called modulation spectral signal representation, which is shown to accurately separate ECG and noise components. The proposed method was tested on synthetic ECG signals corrupted with varying levels of recorded noise and on long-term bedside noisy ECG recordings. Gains over a state-of-the-art wavelet-based denoising algorithm were achieved, particularly for very noisy scenarios. Overall, the proposed algorithm achieved a 61.8% gain in signal-to-noise ratio improvement, a three times reduction in average heart rate measurement error, and a 15% reduction in HRV measurement error relative to the benchmark, thus suggesting that it is an ideal candidate for ECG-based fitness/athletic monitoring applications.