BeamNet: Unsupervised Beamforming for ISAC Systems Under Imperfect CSI

Integrated sensing and communication (ISAC) is expected to be a key enabler for future wireless networks, improving spectral and hardware efficiency by jointly performing radar sensing and wireless communication within a unified framework. This paper proposes BeamNet, an unsupervised deep learning framework for transmit beamforming in dual-function radar-communication systems operating over general fading with imperfect channel state information (CSI). BeamNet maps noisy estimates of the communication and sensing channels to a transmit beamforming vector and is trained end-to-end by maximizing a weighted sum of the communication rate (CR) and sensing rate (SR), thereby learning the CR–SR Pareto frontier without beamforming labels or embedded optimization solvers. Using Rayleigh fading with perfect CSI, we first show that BeamNet reproduces the analytical Pareto-optimal beamforming solutions. We then use BeamNet to characterize, for Nakagami-m and Rician fading, the CR–SR trade-off across a range of fading parameters, and to assess robustness under distribution mismatch between training and test channels. Finally, under imperfect CSI, we demonstrate that BeamNet yields CR–SR trade-offs that are consistently sandwiched between the perfect-CSI and mismatched analytical baselines, outperforming the closed-form beamformer applied to imperfect CSI and recovering part of the performance loss caused by channel estimation errors. These results indicate that unsupervised learning offers a flexible and robust approach to ISAC beamforming in fading environments with imperfect channel knowledge.