There is a tremendous demand from various industries for next-generation 5G networks, which has dramatically increased the need for high data rates and energy efficiency. It is an indisputable fact that next-generation networks should be not only energy-efficient but also resource-efficient. Given the fact that 10% of the current energy consumption in the world is caused by Information and Communication Technology, it is evident that energy-efficiency has become most crucial performance criteria in the next-generation communication techniques. Based on these needs, we previously suggested the hexagonal quadrature amplitude modulation (HQAM) aided spatial modulation (SM) technique (HQAM-SM) to the literature. In this study, we found it appropriate to do this research to further increase the performance of the HQAM-SM scheme through the antenna selection technique and to investigate the effects of antenna selection technique on HQAM-SM. Moving from this point, in this article, rational capacity-optimized antenna selection (COAS), SM, and energy-efficient HQAM techniques are combined, and a new system called COAS-HSM is presented. Hexagonal constellations are to optimize the constellation points to form a hexagonal constellation to minimize the Hamming distance between the constellation points. This layout not only offers better energy efficiency than traditional QAM constellations but also performs almost the same BER performance as QAM at high signal-to-noise ratio (SNR) values. On the other hand, antenna selection algorithms are one of the transmission schemes that have been frequently encountered in the literature in recent years, and that considerably increases the performance of various multiple-input multiple-output (MIMO) communication structures. In particular, the COAS transmission scheme is an intelligent method of selecting transmission antennas over the highest channel amplitudes. Performance analysis of the proposed COAS-HSM technique is carried out in Rayleigh fading channels.