The thermal expansion effect of silicon nanowires (SiNW) in [100], [110] and [111] directions with different sizes is theoretically investigated. At low temperatures, all SiNW studied exhibit a thermal contraction effect due to the lowest energy of the bending vibration mode which has a negative effect on the coefficient of thermal expansion (CTE). The CTE in [110] direction is distinctly larger than the other two growth directions because of the anisotropy of the bending mode in SiNW. Our study reveals that CTE decreases with an increase of the structure ratio γ = length/diameter, and is negative in the whole temperature range with γ = 1.3.