Aging is associated with changes in heart rate (HR), heart rate variability (HRV), and 24-h rhythms in HR. Longevity has been linked to lower resting HR, while a higher resting HR and a decreased HRV were linked to cardiovascular events and increased mortality risk. HR and HRV are often investigated during a short electrocardiogram (ECG) measurement at a hospital. In this study, we aim to investigate the relationship between HR parameters with familial longevity and chronological age derived from continuous ambulatory ECG measurements collected over a period of 24 to 90 hours. We included 73 middle-aged participants (mean (SD) age: 67.0 (6.16) years), comprising 37 offspring of long-lived families, 36 of their partners, and 35 young participants (22.8 (3.96) years). We found no association with familial longevity, but middle-aged participants had lower 24-h HR (average and maximum HR, not minimum HR), lower amplitudes, and earlier trough and peak times than young participants. Associations in HR with chronological age could be caused by the aging process or by differences in environmental factors. Interestingly, middle-aged participants had a less optimal HRV during long-term recordings in both the sleep and awake periods, which might indicate that their heart is less adaptable than that of young participants. This could be a first indication of deteriorated cardiovascular health in middle-aged individuals.