Kangaroo Mother Care (KMC), involving chest-to-chest skin contact between an infant and caregiver, is proven to be an effective intervention for preterm and full-term infants. Accurate monitoring of KMC duration and infant's vital signs during KMC is clinically important. Existing monitoring methods, however, rely on manual efforts and require rigid sensors or wires/electrodes on the infant's body. We propose Joey, a fabric-based approach to continuously monitor KMC duration and two vital signs essential to an infant's well-being: heart rate and respiration rate. Joey is a soft fabric necklace worn by the caregiver. It leverages the transmission of electrocardiogram (ECG) signals across individuals during skin-to-skin contact. With a minimalist fabric sensor structure, Joey measures KMC duration via the presence of mixed ECG signals. It then isolates the infant's ECG from this mixture with a proposed signal extraction algorithm and employs a diffusion-based denoising model to mitigate motion artifacts, enabling reliable inference of infant's vital signs. We fabricate Joey prototypes with off-the-shelf hardware and evaluate its performance with user studies. Results demonstrate that Joey achieves an average F1 score of 96% for KMC duration measurement, and clinically-acceptable accuracy in infant's vital sign estimation with a mean absolute error of 2.3 beats per minute and 2.9 breaths per minute in estimating heart rate and respiration rate. Clinical interviews further confirm the usability of Joey's sensing fabric for infant skin.