Open Access Article
Roya Kavkhani†
a,
Berna Akgenc Hanedar†
*bc,
Kerem Anar
a,
Akile İlknur Şahin
cf,
Annika Johansson
de and
Mehmet Cengiz Onbaşlı
*acf
aGraduate School of Sciences and Engineering (GSSE), Koc University, Rumelifeneri Yolu, Sariyer 34450, Istanbul, Turkey
bDepartment of Physics, Kirklareli University, 39100, Kirklareli, Turkey. E-mail: berna.akgenc@klu.edu.tr
cDepartment of Physics, Koc University, Rumelifeneri Yolu, Sariyer 34450, Istanbul, Turkey. E-mail: bhanedar@ku.edu.tr
dMax Planck Institute of Microstructure Physics, Weinberg 2, 06120 Halle (Saale), Germany. E-mail: annika.johansson@mpi-halle.mpg.de
eHalle-Berlin-Regensburg Cluster of Excellence CCE, Germany
fDepartment of Electrical & Electronics Engineering, Koc University, Rumelifeneri Yolu, Sariyer 34450, Istanbul, Turkey. E-mail: monbasli@ku.edu.tr
First published on 16th January 2026
Topological insulators (TIs) such as Sb-doped Bi2Te3 and Bi2Se3 exhibit promising phenomena for advanced spintronics. While previous studies explored isolated doping levels; a systematic understanding of how Sb concentration influences topological behavior, Rashba-type spin splitting, and surface state formation is lacking. Here, we use density functional theory to investigate the structural, electronic, topological and transport properties of (Bi1−xSbx)2Te3 and (Bi1−xSbx)2Se3 thin films across 0 ≤ x ≤ 1. We identify pronounced Rashba spin splitting in Bi2Te3 at x = 0.5, 0.6, and 0.9 with in-plane helical spin textures. We identified the orbital origins of topological surface states and demonstrate that band inversion persists across the Sb doping range. At x = 0.2, 0.4, and 0.8, calculated surface electron mobilities are consistent with experiments and increase an order of magnitude over Sb2Te3, with minimal impact on bulk mobilities. These insights advance our understanding of TIs for spintronic and quantum device applications.
Beyond the fundamental topological surface states, another SOC-driven phenomenon plays a crucial role in TI functionality:6,7 Rashba-type spin splitting, which emerges under broken inversion symmetry.8 Unlike bulk band inversion, Rashba splitting is typically induced by surface reconstruction9 or interface effects,10–12 resulting in momentum-dependent spin textures that can be systematically controlled through chemical doping.13–15
While the effects of individual dopants on topological insulators have been widely studied,16–19 the interplay between doping-induced modifications and Rashba-type effects on surface state properties remains largely unexplored. Previous studies have primarily focused on isolated doping scenarios, neglecting a critical aspect. When dopant atoms exhibit correlations or mutual interactions, the physical behavior fundamentally changes, and Rashba effects become significantly more pronounced.
This correlated doping regime offers control over TI properties, potentially enabling fine-tuning of both topological protection and spin–orbit phenomena in ways not achievable through conventional random doping approaches.14,15,20 This research gap is particularly crucial, as optimal TI functionality likely depends on a delicate balance between topological robustness and tunable spin–orbit effect, where dopant correlations and enhanced Rashba contributions could serve as a critical pathway for achieving advanced quantum control. Such a balance can only be realized through systematic compositional engineering that incorporates dopant interactions and their amplified impact on surface state manipulation.
Among known TIs, Bi2Te3
21 and Bi2Se3 have emerged as benchmark systems due to their well-defined topological properties,22–24 layered crystal structures, and strong SOC. These materials have been widely studied both theoretically and experimentally, serving as model platforms for exploring the manifestation of topological surface states (TPSS) and their manipulation via external fields and chemical modifications.4,25,26 A common strategy to modulate their electronic structure is isovalent substitution, particularly by introducing antimony (Sb) in place of bismuth (Bi). Sb shares the same valence electron configuration with Bi, making it a chemically compatible dopant that allows for electronic tuning without causing severe distortions to the crystal lattice.27,28
Sb substitution in Bi2Te3 and Bi2Se3 has been extensively explored through experimental and density functional theory (DFT) studies. For 20% Sb-doped Bi2Se3 thin films, Sb shifts the Dirac cone below the Fermi level and enhances surface–bulk hybridization.29 In bulk (Bi1−xSbx)2Te3 (0 ≤ x ≤ 1), DFT revealed that Sb tuning modifies the Dirac cone, spin–orbit coupling strength, and defect formation energies.30 Experimental and DFT studies on ultrathin Bi1.5Sb0.5Te1.7Se1.3 showed a thickness-dependent hybridization gap and topological phase transitions.31 A theoretical DFT-based study on 3D Bi2Se3-type topological insulators (Bi2Te3, Bi2Se3, Sb2Te3) developed an effective four-band k·p Hamiltonian to describe bulk electronic structures and surface Dirac states, providing a foundation for modeling topological features in these materials.32 Another DFT investigation on Bi2Se3 nanosheets with Sb2Se3 and SbBiSe3 surface capping layers analyzed the role of interface asymmetry and biaxial strain (±5%), showing that Sb-based capping opens a band gap by breaking inversion symmetry, while strain can restore or tune the topological surface states and spin-momentum locking.33
While Sb doping27,28,34 has been extensively used to shift the Fermi level, the full extent of its impact on the electronic topology, including the evolution of band inversion, the emergence or suppression of Rashba-type splitting, and the modification of SOC strength, is not yet fully clarified. In particular, previous studies have often focused on isolated doping levels or specific material systems, lacking a comprehensive and systematic investigation across a continuous doping range. Additionally, the surface versus bulk character of emerging states under doping has not been thoroughly analyzed, leaving questions about the robustness of topological protection under high substitution levels.
Recent work on (Bi1−xSbx)2Te3 demonstrated that Sb concentration can be used to suppress bulk conduction and reveal surface-dominated transport at x ≈ 0.5, enabling effective gating and ambipolar field effects.34 At the same time, giant and tunable Rashba-type spin splitting has been observed in polar Bi-based systems such as BiTeI and surface-doped Bi2Se3, where strong SOC and inversion asymmetry induce spin-resolved electronic states suitable for spintronic applications.35,36 Despite these advances, the systematic emergence and evolution of Rashba-type features across the full Sb doping range remain unresolved. Moreover, the competition and possible coexistence between Rashba-split states and topological surface states in doped Bi2Te3 and Bi2Se3 are not yet fully understood.
In this study, we address these critical gaps by performing a systematic DFT analysis on two-dimensional (2D) Sb-doped Bi2Te3 and Bi2Se3 across a wide doping range, from pristine compound to the fully Sb-substituted case. By examining the evolution of the electronic band structure, orbital-projected contributions, and symmetry-breaking effects, we aim to clarify how Sb doping modifies key topological and spin-related features. Special attention is devoted to identifying the existence of band inversion features, assessing the onset and strength of Rashba-type spin splitting, and examining how these topological features respond to varying doping levels. This work provides a detailed theoretical framework to understand the doping-dependent transformation of topological insulators, offering insights that are essential for their integration into next-generation spintronic and quantum technologies. We calculate the electron effective masses, bulk and surface electronic mobilities as a function of Sb concentration. Sb-alloyed Bi2Te3 derivatives exhibit strong spin–orbit coupling, giving rise to Rashba-type spin splitting and high spin-charge conversion. To describe this situation, semiclassical Boltzmann transport calculations combined with DFT-derived parameters provide an efficient route to link SOC. In this work, we focus on the (Bi0.4Sb0.6)2Te3 composition, analyzing its Rashba-split bands and transport properties using the Rashba model.37 In particular, existing Rashba-type splitting and spin-momentum locking in these systems highlights their potential utility in spin field-effect transistors, spin–orbit torque-based memory, and topological quantum computing architectures.
To cross-check Rashba-related features and corresponding transport response we can use the Hamiltonian and DFT-derived parameters as an input to the Boltzmann Transport Equation (BTE), validating that the observed spin splitting behavior originate from structural inversion asymmetry. The velocity and spin expectation values were computed over the Fermi surface to calculate the corresponding transport tensors, including Edelstein and conductivity tensors.43 A numerical solver was implemented using single-particle Hamiltonian which includes Rashba spin–orbit coupling and warping effect, expressed as:
![]() | (1) |
The emergence of TI properties in Bi2Te3 and Bi2Se3 is fundamentally governed by two key mechanisms: band inversion and band gap closing, both of which occur as a function of thickness. In their bulk form, these materials exhibit strong SOC, which induces an inverted band structure and supports the formation of gapless Dirac surface states. However, when these compounds are confined to ultrathin films, the interaction between the top and bottom surfaces becomes significant. This inter-surface hybridization opens a gap at the Dirac point, effectively suppressing the TI phase and rendering the system a trivial insulator.45 This hybridization gap dominates the electronic structure in films with very small thicknesses, where the surface wavefunctions strongly overlap. With increasing film thickness, the interaction between opposite surfaces weakens, allowing the emergence of band inversion. Once the hybridization gap weakens sufficiently, the system undergoes a topological phase transition, restoring the gapless Dirac surface states that are the hallmark of TIs. The critical thickness at which this transition occurs differs between Bi2Te3 and Bi2Se3, owing to intrinsic differences in their electronic band structures and the strength of their SOC.46–48
The first-principles calculations46 confirm that the decay length of surface states is approximately 1 nm, resulting in strong hybridization in 1QL films. This hybridization effect becomes progressively weaker beyond 2QL, leading to reduced gap opening. Experimentally, this manifests as the emergence of a single Dirac cone surface state at 2QL.46
m) and a hexagonal conventional cell, with the notable exception of Sb2Se3, which naturally crystallizes in an orthorhombic phase (Pnma).21 While the first three compounds are well-established TIs, Sb2Se3 is a trivial insulator in its equilibrium phase.45 However, theoretical and experimental studies have demonstrated that Sb2Se3 can undergo a topological phase transition under specific conditions such as via external strain49 or by forming a heterointerface with a topological phase like Bi2Se3
50 where interfacial hybridization induces nontrivial surface states.
Fig. 2 illustrates the side-view atomic structures of the 5QL slabs used in this study for both (Bi1−xSbx)2Te3 (BST) and (Bi1−xSbx)2Se3 (BSS) systems. As shown in Fig. 2(a), each QL consists of five atomic planes arranged in a Te(Se)–Bi–Te(Se)–Bi–Te(Se) sequence, with an inter-QL spacing.
To systematically investigate the effect of Sb concentration on structural and electronic properties, we substituted Bi atoms with Sb in 0.1 increments, generating eleven configurations from x = 0 to x = 1.0. The goal was to identify the doping level at which TI behavior emerges, as shown in Fig. 2(a)–(k). For each doping level, the configuration with the lowest formation energy was selected to ensure structural stability and physical relevance in subsequent analyses (SI Table S1). All structures were modeled within the same rhombohedral crystal framework, using consistent substitution patterns to ensure structural uniformity across the series. This controlled setup allows for a clear comparison of doping effects without interference from phase transitions or distortions. Electronic band structure and spin texture were calculated along the high-symmetry path
–
–
in the two-dimensional surface Brillouin zone, as illustrated in Fig. 2(l).
To evaluate structural stability, we computed both the formation energy (Efor) and cohesive energy (Ecoh) per atom using the following expressions:
![]() | (2) |
![]() | (3) |
Fig. 3 shows the systematic evolution of the in-plane lattice parameters and the out-of-plane lattice constant, represented by the 5QL slab thickness, as a function of Sb concentration for both BST and BSS systems. In BST, the in-plane and out-of-plane lattice constants a and c decrease from 4.302 and 49.658 Å at x = 0.1 to 4.247 and 48.828 Å at x = 0.9. A similar pattern emerges in BSS, with a and c shrinking from 4.107 and 45.792 Å to 4.025 and 45.476 Å. This behavior arises from the smaller atomic radius of Sb compared to Bi, leading to a lattice shrinkage upon substitution. The nearly linear variation of lattice parameters with composition in both systems is consistent with Vegard's law. Additionally, our ab initio molecular dynamics (AIMD) simulation results also demonstrate that these materials remain thermally stable at room temperature as shown in Fig S2 within the SI.
Tables 1 and 2 show that interlayer chalcogen–chalcogen (Te–Te or Se–Se) distances do not follow a monotonic trend. In BST, Te–Te separations fluctuate between 2.854 and 2.925 Å, while in BSS, Se–Se distances vary from 2.669 to 2.760 Å. These deviations likely result from local structural relaxations or distortions induced by Sb substitution. Importantly, both Efor and Ecoh decrease progressively with increasing Sb concentration, suggesting a decline in thermodynamic stability. Although the extent of destabilization is comparable between systems, the lower absolute cohesive energy in BST is consistent with the softer bonding nature of telluride systems. These results indicate that while Sb substitution is energetically feasible, higher doping levels reduce lattice cohesion and compromise bonding integrity.
| Compound | Efor (eV) | Ecoh (eV) | dint-ave (Å) | EΓ-gap (eV) | αR (eV Å) |
|---|---|---|---|---|---|
| Bi2Te3 | −0.343 | 3.121 | 2.870 | 0.009 | — |
| (Bi0.9Sb0.1)2Te3 | −0.329 | 3.110 | 2.880 | 0.030 | — |
| (Bi0.8Sb0.2)2Te3 | −0.314 | 3.101 | 2.877 | 0.069 | — |
| (Bi0.7Sb0.3)2Te3 | −0.299 | 3.091 | 2.889 | 0.031 | — |
| (Bi0.6Sb0.4)2Te3 | −0.285 | 3.081 | 2.854 | 0.009 | — |
| (Bi0.5Sb0.5)2Te3 | −0.270 | 3.071 | 2.892 | 0.082 | 0.328 |
| (Bi0.4Sb0.6)2Te3 | −0.257 | 3.061 | 2.920 | 0.131 | 2.098 |
| (Bi0.3Sb0.7)2Te3 | −0.243 | 3.052 | 2.912 | 0.031 | — |
| (Bi0.2Sb0.8)2Te3 | −0.229 | 3.043 | 2.925 | 0.024 | — |
| (Bi0.1Sb0.9)2Te3 | −0.215 | 3.034 | 2.905 | 0.047 | 0.719 |
| Sb2Te3 | −0.202 | 3.025 | 2.925 | 0.004 | — |
| Compound | Efor (eV) | Ecoh (eV) | dint-ave (Å) | EΓ-gap (eV) | αR (eV Å) |
|---|---|---|---|---|---|
| Bi2Se3 | −0.513 | 3.372 | 2.682 | 0.010 | — |
| (Bi0.9Sb0.1)2Se3 | −0.495 | 3.358 | 2.694 | 0.020 | — |
| (Bi0.8Sb0.2)2Se3 | −0.476 | 3.343 | 2.718 | 0.064 | 0.304 |
| (Bi0.7Sb0.3)2Se3 | −0.458 | 3.330 | 2.711 | 0.034 | — |
| (Bi0.6Sb0.4)2Se3 | −0.440 | 3.317 | 2.702 | 0.023 | — |
| (Bi0.5Sb0.5)2Se3 | −0.422 | 3.303 | 2.736 | 0.035 | — |
| (Bi0.4Sb0.6)2Se3 | −0.405 | 3.290 | 2.738 | 0.079 | — |
| (Bi0.3Sb0.7)2Se3 | −0.388 | 3.278 | 2.757 | 0.067 | — |
| (Bi0.2Sb0.8)2Se3 | −0.371 | 3.265 | 2.731 | 0.080 | — |
| (Bi0.1Sb0.9)2Se3 | −0.354 | 3.253 | 2.760 | 0.079 | — |
| Sb2Se3 | −0.337 | 3.241 | 2.771 | 0.078 | — |
![]() | ||
| Fig. 4 Electronic band structures of 5QL (a) (Bi1−xSbx)2Te3 and (b) (Bi1−xSbx)2Se3 for varying Sb concentrations (x = 0 to 1), showing the evolution of surface states and band dispersion with doping. | ||
In the BST system [Fig. 4(a)], we observe a progressive evolution of surface states with increasing Sb concentration. Notably, a Dirac gap opens at the Γ-point, which coexists with Rashba-type splitting of surface bands around this region. The Rashba splitting exhibits a non-monotonic trend with increasing Sb concentration. The most pronounced splitting occurs at x = 0.6, which also corresponds to the maximum Dirac gap opening at the Γ-point (see Table 1). The finite bandgap opening can thus be attributed to residual coupling between top and bottom surface states in thin films, where Sb-induced local structural asymmetry modifies the decay length and overlap of surface Bloch states, leading to inter-surface hybridization and the formation of a Dirac gap at Γ even without breaking time-reversal symmetry.33 Additional peaks in Rashba-type separation are identified at x = 0.5 and x = 0.9, suggesting that structural inversion asymmetry (SIA) is modulated in a non-linear manner by Sb doping. This behavior likely originates from complex lattice relaxations and local potential gradients induced by Sb substitution.
In contrast, the BSS system [Fig. 4(b)] exhibits a notably different trend. Despite following identical doping configurations, significant band splitting is observed only at x = 0.2. For all other doping levels, the Dirac-like surface states remain essentially degenerate near the Γ-point. This disparity between BST and BSS highlights the material-specific sensitivity of Rashba splitting to doping, which can be attributed to differences in lattice stiffness, intrinsic SOC strength, and atomic hybridization. Specifically, the heavier Te atom in BST contributes to stronger SOC, amplifying spin-splitting effects.
To provide direct evidence of Rashba-type interaction, we examine the spin-resolved band structure and spin textures for the x = 0.6 case in BST, where the band splitting is maximal (Fig. 5). Fig. 5(a) shows the spin-split band structure near the Γ-point, with clear formation of inner and outer bands, resembling canonical Rashba behavior. Fig. 5(b) depict the in-plane isoenergetic spin textures for the outer and inner bands at a constant energy. The outer band exhibits counterclockwise (CCW) spin rotation, while the inner band shows clockwise (CW) rotation, confirming helical spin-momentum locking typical of Rashba systems.35,51
To further support this, Fig. 5(e) illustrates the spin textures at E = 0.3 eV, where both the inner and outer bands remain visible, reinforcing the non-degenerate, spin-polarized nature of the states. These results are well described by the Rashba Hamiltonian, which captures the spin splitting induced by structural inversion asymmetry and spin–orbit interaction:32,52
![]() | (4) |
Diagonalizing this Hamiltonian yields the energy dispersion relation:
![]() | (5) |
![]() | (6) |
| ΔE = 2αRk | (7) |
As illustrated in Fig. 4(a) for x = 0.6, the Rashba effect results in a characteristic band structure featuring a momentum offset (k0 = Δk/2) and a Rashba energy ER, defined as the energy difference between the band minimum and the degeneracy point at k = 0. The Rashba parameter αR can be conveniently expressed in terms of ER and k0 as:
![]() | (8) |
This pronounced spin splitting, particularly evident in BST at intermediate Sb concentrations, arises from the strong spin–orbit coupling and structural inversion asymmetry, manifesting as Rashba-type band splitting with helical spin textures, consistent with the spin-split surface states observed in systems such as Au(111) due to broken inversion symmetry and SOC.53
A quantitative analysis of the extracted αR values (provided in Tables 1 and 2) reveals a peak at x = 0.6 in BST, where αR = 2.098 eV Å, indicating a remarkably strong linear Rashba interaction. In contrast, the BSS system shows a significantly weaker response, with a maximum value of only αR = 0.304 eV Å at x = 0.2. This disparity highlights the stronger structural inversion asymmetry (SIA) and spin–orbit coupling in Te-based systems compared to their Se-based counterparts under similar doping configurations. The pronounced Rashba spin splitting observed in BST, especially at intermediate Sb doping, parallels the large and tunable Rashba effect reported in Bi2Se3-based 2DEGs,36 highlighting how intrinsic SIA and strong SOC can induce spin-polarized, non-degenerate states even without external gating, offering a robust platform for spintronic applications. The calculated dipole moment (μz) and the LDOS distribution (SI Table S2 and Fig. S3) both show polarity reversal and charge asymmetry at x = 0.6, confirming strong SIA, which in turn leads to a pronounced Rashba splitting.
The observed Rashba parameter in BST not only exceeds that of BSS, but also surpasses many well-known Rashba-active materials. For instance, quantum-confined near-surface states in K-deposited Bi2Se3 yield αR = 0.79 ± 0.03 eV Å (RB1) and αR = 0.35 eV Å (RB2),54 while magnetotransport studies on epitaxial Bi2Te3 films report αR = 1.58 eV Å.13 Additionally, metallic systems such as Bi(111),55 Ir(111),56 and PtPb4
57 exhibit lower values of approximately 0.55, 0.35, and 1.0 eV Å, respectively. Even the Bi/Ag(111) interface, engineered to maximize Rashba splitting, reaches approximately 3.0 eV Å under ideal conditions.58 The fact that BST approaches these engineered values through doping alone demonstrates the effectiveness of Sb substitution in enhancing SIA. The pronounced Rashba splitting in BST, particularly at intermediate Sb concentrations, reflects robust spin–momentum locking and strong SOC, hallmark features of topological materials. These characteristics position Sb-doped Bi2Te3 as a promising platform for spintronic technologies, including spin field-effect transistors (SpinFETs), spin–orbit torque (SOT)-based memory devices, and future topological quantum computing architectures.
![]() | (9) |
Due to the intrinsic anisotropy of topological insulator (TI) materials, calculations were performed independently along the in-plane x- and y-directions. To enable directional strain application, the original rhombohedral crystal structures of (Bi1−xSbx)2Te3 with x = 0, 0.2, 0.6, 0.8, and 1 were transformed into orthorhombic supercells (see Fig. S4(a) in the SI). After structural relaxation, a series of uniaxial strains ranging from −6% to +6% (in 2% increments) were applied along each direction. The total energy corresponding to each strain state was computed using first-principles calculations, and the resulting energy–strain relationship was fitted using a quadratic function.
The elastic modulus C was obtained from the second derivative of the total energy with respect to strain as shown in Fig. S4(b–f) in the SI:
![]() | (10) |
The deformation potential constant E1 was extracted by tracking the shift in the energy of the conduction band minimum (CBM) of both surface and bulk states as a function of strain, shown in Fig. S5 and S6 in the SI. The vacuum level was used as a reference to determine the absolute positions of the band edges under strain. The energy shifts were linearly fitted, and the slope of the fit yielded the deformation potential:
![]() | (11) |
The carrier effective mass m* was calculated by fitting the curvature of the electronic bands near the CBM or VBM using the parabolic approximation:
![]() | (12) |
High-resolution band structure data along the x direction were analyzed using VASPkit. When necessary, the density-of-states effective mass was calculated using the geometric mean:
.
In this study, the electron mobilities and related transport parameters of Sb-doped Bi2Te3 were analyzed across the doping range x = 0.0 to x = 1.0 at room temperature (298 K), with a particular focus on surface and bulk contributions. Hole mobilities were not evaluated separately for surface and bulk states due to the difficulty in distinguishing the corresponding bands. As shown in Table 3, surface mobility μSurface dominates over bulk mobility μBulk across all compositions, confirming the expected behavior in topological insulator systems where robust surface states play a key role in conduction. A notable enhancement in surface mobility is observed around x = 0.4 and x = 0.8 reaching 1.04 × 105 and 9.76 × 104 cm2 V−1 s−1, respectively, whereas the bulk mobilities remain significantly lower (<1.1 × 104 cm2 V−1 s−1). This behavior correlates with the minimized effective mass and moderate deformation potential values in this doping range, indicating reduced carrier-phonon scattering and enhanced carrier transport along the surface channels.
| Compound | ESurf1 (eV) | EBulk1 (eV) | C2D (N m−1) | mSurfa (m0) | mBulka (m0) | md,Surf (m0) | md,Bulk (m0) | μSurf (×104 cm2 V−1 s−1) | μBulk | Method |
|---|---|---|---|---|---|---|---|---|---|---|
| a The film was grown on a buffer layer of In2Se3/(Bi0.5In0.5)2Se3 on an Al2O3 substrate. | ||||||||||
| Bi2Te3 | 8.18 | 7.35 | 253.03 | 0.05 | 0.31 | 0.04 | 0.24 | 3.38 | 0.13 | In this study |
| (Bi0.8Sb0.2)2Te3 | 4.25 | 3.61 | 260.81 | 0.06 | 0.21 | 0.05 | 0.20 | 9.03 | 1.03 | In this study |
| (Bi0.6Sb0.4)2Te3 | 5.94 | 5.17 | 266.48 | 0.04 | 0.30 | 0.04 | 0.33 | 10.43 | 0.21 | In this study |
| (Bi0.2Sb0.8)2Te3 | 5.98 | 5.39 | 280.29 | 0.04 | 0.42 | 0.04 | 0.45 | 9.76 | 0.11 | In this study |
| Sb2Te3 | 6.74 | 6.09 | 292.46 | 0.13 | 0.27 | 0.07 | 0.22 | 1.43 | 0.27 | In this study |
| Bi2Te3 | — | — | — | — | — | — | — | 0.60 | 0.09 | ME61 |
| Bi2Te3 | — | — | — | — | — | — | — | — | 0.05 | MBE62 |
| Bi2Se3 (uncapped)a | — | — | — | — | — | — | — | 1.60 | — | MBE63 |
| Bi2Se3 (Se-capped)a | — | — | — | — | — | — | — | 0.36 | — | MBE63 |
| Bi2Se3 (MoO3-capped)a | — | — | — | — | — | — | — | 0.33 | — | MBE63 |
| Graphene | — | — | — | — | — | — | — | — | 1.00 | CVD64 |
| Graphene | — | — | — | — | — | — | — | — | 35.00 | CVD65 |
Our theoretical predictions, based on DFT calculations at 0 K for ideal 2D (Bi1−xSbx)2Te3 systems, demonstrate surface mobilities that are significantly higher than typical experimental values. For instance, we find that the calculated surface mobility peaks at x = 0.4, exceeding 105 cm2 V−1 s−1 which are consistent with the trends reported in experimental studies. For example, exfoliated Bi2Te3 ultrathin films have shown mobilities of approximately 6030 cm2 V−1 s−1 at 2 K,61 considerably exceeding those obtained from MBE-grown (521 cm2 V−1 s−1)62 and chemically synthesized samples. Although our theoretical results inherently overestimate absolute mobility values66 due to the absence of phonon scattering, defects, and substrate effects; they capture the non-monotonic doping dependence, with maximum mobility occurring near x = 0.4–0.6. This trend is in good agreement with previous experimental observations of enhanced Rashba spin splitting and surface-dominated conduction in this doping regime.13,67–69 The observed dependence on Sb content further reflects the underlying changes in electronic band structure, effective mass, and electron–phonon coupling, all of which are sensitively captured in first-principles deformation potential calculations. Thus, while quantitative differences remain between theory and experiment, the agreement in doping-dependent mobility behavior strongly supports the predictive power of DFT-based models for guiding the optimization of topological materials. When compared to high-mobility 2D materials such as graphene, where mobilities range from 10
000 cm2 V−1 s−1 (ref. 64) in ambient conditions to 350
000 cm2 V−1 s−1 for CVD-grown ultraclean samples,65 Sb-doped Bi2Te3 films show competitive surface transport properties while maintaining topologically protected states. These results demonstrate the tunability of surface transport in (Bi1−xSbx)2Te3 through controlled doping and provide a pathway for engineering high-performance topological devices, particularly in regimes where surface conduction can be maximized and bulk contributions can be minimized.
For doping levels x = 0.6, the DFT-derived Rashba parameters correspond to a significant spin splitting. The BTE results reveals that the largest Edelstein response, which describes the efficiency and the direction of charge-to-spin conversion, and charge conductivity (shown in SI) occur near at x = 0.6, consistent with the DFT-predicted maximum Rashba-splitting at similar doping levels. Edelstein response quantifies how efficiently an applied charge current generates a net spin density, providing a direct measure of charge-to-spin conversion in Rashba systems, which is essential for spintronic applications. This confirms that moderate Sb substitution optimizes the balance between spin–orbit interaction strength and band inversion. Furthermore, the similarity between the analytical Rashba model and the DFT band structure confirms that the simplified approach captures the essential physics of spin–orbit-driven band structures while enabling much faster computation transport tensors compared to full DFT calculations. The observed trends can be traced to the strength of Rashba coupling and the nature of impurity scattering. Increasing αR enhances Rashba splitting, which enhances the in-plane Edelstein response. The numerical results demonstrate that spin–charge conversion peaks near x = 0.6, instead of DFT-derived parameters within the semiclassical BTE framework. While the model neglects electron–phonon coupling and interfacial roughness effects, it provides a reliable first-order description of spin–orbit-driven transport at moderate impurity concentrations.
To determine whether the weakening of band inversion manifests in the topological properties, the contributions of surface and bulk atoms to the formation of surface states (SS) were examined. The crystal structures were spatially partitioned: the top and bottom QLs were designated as the surface region, while the central three QLs represented the bulk. This classification enables a focused evaluation of the Dirac cone's spatial origin, based on the assumption that surface states predominantly arise from the outermost atomic layers.
Fig. 7(a) displays the projected p-orbital band structures for the surface and bulk regions of BST at x = 0.1, 0.5, and0.9. The corresponding results for other doping levels in both BST and BSS are provided in Fig. S8 and S9 within the SI for BST and BSS, respectively. As shown, the surface atoms clearly dominate the formation of surface states at all concentrations. This finding is further corroborated by Fig. 7(b), which illustrates the projections onto the bulk atoms, revealing minimal or negligible contribution to the SS.
This distinction underscores a defining characteristic of topological insulators: while the bulk remains insulating, the metallic surface states are confined to the system's boundaries. The consistent absence of significant bulk contribution across all doping levels affirms that the Dirac-like surface states originate from topological protection enforced by crystal symmetry and strong spin–orbit coupling.
To better understand the transport behavior of the studied materials, carrier mobilities were calculated separately for surface and bulk states based deformation potential theory. The results reveal a clear and systematic distinction between surface and bulk mobilities across all compositions, with surface mobilities consistently exceeding their bulk counterparts by a significant margin. This indicates that charge transport is predominantly governed by surface states rather than bulk conduction. The enhanced mobility at the surface suggests more favorable conditions for carrier movement, likely due to reduced scattering and a distinct band structure. In contrast, the bulk region exhibits much lower mobility values, reflecting limited carrier transport efficiency. These findings consistently highlight the dominant role of surface channels in the overall electronic transport behavior.
Furthermore, we demonstrated that the topological surface states (TSS) persist across the full doping range, albeit with a gradual weakening of band inversion, indicating a continuous rather than abrupt topological phase evolution. Spin texture analysis confirmed helical spin-momentum locking and clear signatures of inner and outer spin-split bands, which are typical of Rashba systems, substantiating the coexistence of Rashba-type interaction and topological protection in these materials.
This study offers new perspectives on Sb-induced modifications in well-known topological insulators but also provide a foundation for controlling key properties such as Rashba spin splitting and the stability of topological surface states across varying compositions. Our carrier mobility analysis further confirms that surface transport channels dominate over bulk conduction at all doping levels, reinforcing the critical role of surface states in practical charge transport. Together, these findings position (Bi1−xSbx)2Te3 and (Bi1−xSbx)2Se3 as highly tunable platforms, paving the way for advanced applications in spin-orbitronics and topological quantum computing.
Data for this article, including Fig. 2 are available at VESTA at https://jpminerals.org/vesta/en/download.html.
Data for this article, including Fig. 1 and 4–7 are available at VASP (Vienna Ab initio Simulation Package) at https://www.vasp.at/.
The illustration shown in Fig. 5(d) was drawn using Adobe Illustrator at https://www.adobe.com/products/illustrator.html.
The videos used in the supplementary information (SI) were recorded with ASE (Atomic Simulation Environment) at https://ase-lib.org/about.html.
The data supporting this article have been included as part of the SI. Supplementary Information provides details on stable atomic configurations, additional electronic band structures, Rashba-related transport (Edelstein tensor and conductivity), dipole moment and LDOS analyses, deformation-potential calculations, AIMD thermal stability tests, and surface-bulk orbital projections supporting the main text. Supplementary information is available. See DOI: https://doi.org/10.1039/d5nr03682h.
Footnote |
| † These authors contributed equally to this work as co-first authors. |
| This journal is © The Royal Society of Chemistry 2026 |