A novel subcell limiting for high-order entropy-stable DGSEM to prevent carbuncle phenomenon

High-order discontinuous Galerkin spectral element method (DGSEM) often encounter shock instability or carbuncle phenomenon when handling compressible flow problems with discontinuities. To solve this problem,  a novel subcell limiting strategy satisfying discrete entropy inequality is proposed. Firstly, an entropy-stable shock-capturing scheme is developed based on sign-preserving interpolation of entropy variables and an entropy stable flux. Then, the shock capturing scheme is  used in subcell limiting for high-order entropy-stable DGSEM on Legendre-Gauss solution points to prevent carbuncle phenomenon. The entropy-stable DGSEM with the subcell entropy-stable shock capturing scheme is in fact a hybrid scheme. The discrete conservation laws and discrete entropy inequality are proved theoretically for the proposed hybrid scheme. Numerical investigations show that the proposed method satisfies both discrete conservation laws and entropy stability, achieves high resolution in smooth regions, has strong shock-capturing capability, and can effectively prevent  carbuncle phenomenon.