Security Enhancement of Playfair Cipher Using Modified Blum Blum Shub Algorithm and Keystream Values

Abstract The Playfair cipher operates as a symmetric en-cryption method, simultanously encrypting pairs of letters. This research aims to enhance the Playfair cipher’s security by incorporating a modified Blum Blum Shub algorithm and utilizing Keystream values. The Blum Blum Shub algorithm undergoes modification through the introduction of four Blum prime numbers, complicating prime factorization. These primes are employed to generate random numbers, subsequently forming a key by mapping the equivalent characters of the generated sequences. In this investigation, we ensure secure key swapping, eliminating the necessity for filler characters when combining related characters in to bigrams. Additionally, we obscure the relationship between plaintext and ciphertext bigrams by altering the Playfair cipher’s encryption mechanism. Notably, the recipient does not directly receive the key; instead , it independently generates the same key as the sender, resolving the key exchange challenge. The proposed algorithm undergoes evaluation on an HP computer using Matlab software, assessing its performance in terms of the avalanche effect, frequency analysis, key generation, key exchange, and resistance against brute force attack. The simulation results demonstrate that the proposed algorithm yields a high avalanche effect ratio. It generates a complex key, challenging to predict , and requires a substantial amount of time for cryptanalysis attacks to crack. A slight modification in a single plaintext character results in an average avalanche effect of 80%. Consequently, the proposed approach is more secure than the extended algorithm.