Introduction: Physical activities in children have various educational effects, particularly on non-cognitive skills, which have been recently emphasized. The Organization for Economic Co-operation and Development (OECD) considers non-cognitive abilities as social and emotional skills (SES) and has proposed a measurement scale; however, research on their relationship with physical activity and physical fitness is insufficient. Therefore, this study examined the impact of physical fitness improvement during young childhood on SES during the elementary school years, further investigated the mediating effects of physical activity preferences and physical fitness during the elementary school years on SES. Methods: Participants were 117 fifth-grade elementary school students who participated in annual physical fitness tests during young childhood and remained traceable until 2024. We conducted a physical fitness test and the OECD's SES Scale; physical activity preference was assessed using a questionnaire. Physical fitness changes during young childhood were set as the explanatory variable, physical fitness and physical activity preferences at the 5th-grade level as the mediating variables, and each SES factor in 5th-grade as the dependent variable. Following Baron and Kenny's standard procedure (Baron & Kenny, 1986), we first examined the direct effect of the explanatory variable on the dependent variable using regression analysis. Second, we examined the mediating path from the explanatory variable to the mediating variable and from the mediating variable to the dependent variable. Subsequently, a mediation analysis was conducted using a structure equational modeling. Results: The direct effects of changes in the 25m run and ball throw on SES were confirmed. The SES factors with significant effects were “assertiveness,” “energy,” “sociability,” “co-operation,” and “empathy.” In confirming mediating pathways, regression analysis revealed that changes in ball throw during young childhood showed significant regression coefficients for 5th-grade physical fitness and physical activity preference. The p-values for the regression coefficients of the 25m run were 0.055 and 0.119, respectively. Regression analysis for each SES factor revealed significant regression coefficients for physical activity preference in relation to “assertiveness,” “sociability,” and “cooperation.” Significant regression coefficients were obtained for both 5th-grade physical activity preference and physical fitness for the “energy” factor. Finally, A significant mediating effect of physical activity preference was confirmed for all factors except “empathy”; 5th-grade physical fitness also had a significant mediating effect for “energy.” Discussion: Improvements in 25-m run and ball throw during young childhood were effective in enhancing SES by the fifth grade. Additionally, enhanced physical activity preference was an important mediating factor for SES improvement.