A millimeter-scale periodic structure is created in a polyelectrolyte hydrogel by the rapid-heterogeneous swelling process, and is frozen by the polyion complexation of the polyelectrolyte network with the oppositely charged, semi-rigid polyelectrolyte. The hydrogel is synthesized from a cationic monomer, N-[3-(N,N-dimethylamino)propyl] acrylamide methyl chloride quaternary (DMAPAA-Q), in the presence of a small amount of the oppositely charged poly(2,2′-disulfonyl-4,4′-benzidine terephthalamide) (PBDT) that has a semi-rigid nature. During the swelling process, surface creasing due to the large mismatching of swelling degree between the surface layer and the inner one of the poly DMAPAA-Q (PDMAPAA-Q) gel occurs, which induces highly oriented semi-rigid PBDT molecules along the tensile direction of the crease pattern. To accompany the evolution of surface creasing, a lattice-like periodic birefringence pattern is formed, which is frozen permanently by the strong polyion complex formation, even after the surface instability pattern of the gel disappears completely throughout the dynamic coalescence. In this work we rationally clarified that formation of such a long-range ordered non-equilibrium structure in the polyelectrolyte hydrogel by the rapid-heterogeneous swelling process requires the following three indispensable conditions: (i) swelling-induced surface creasing; (ii) polyion complex formation; and (iii) a semi-rigid or rigid dopant. This sort of non-equilibrium structure formation mechanism may help understand how biomacromolecules that are rigid polyelectrolytes, such as deoxyribonucleic acid, microtubules and actin filaments, form rich architectures during the growth of biological organs.