Sima Qian was born and grew up in Longmen, near present-day Hancheng, Shaanxi. He was raised in a family of astrologers. His father, Sima Tan, served as the Prefect of the Grand Scribes of Emperor Wu of Han (Emperor "Han Wudi"). His main responsibilities were managing the imperial library and maintaining or reforming the calendar. Due to the intensive training given by his father, by the age of ten, Sima Qian was already well versed in old writings. He was the student of the famous Confucians Kong Anguo and Dong Zhongshu. At the age of twenty, with the support of his father, Sima Qian started a journey throughout the country, collecting useful first-hand historical records for his main work, Shiji. The purpose of his journey was to verify the ancient rumors and legends and to visit ancient monuments, including the renowned graves of the ancient sage kings Yu and Shun. Places he had visited include Shandong, Yunnan, Hebei, Zhejiang, Jiangsu, Jiangxi and Hunan.

After his travels, he was chosen to be a Palace Attendant in the government whose duties were to inspect different parts of the country with Emperor Han Wudi. In 110 BC, at the age of thirty-five, Sima Qian was sent westward on a military expedition against some "barbarian" tribes. That year, his father fell ill and could not attend the Imperial Feng Sacrifice. Suspecting his time was running out, he summoned his son back to complete the historical work he had begun. Sima Tan wanted to follow the Annals of Spring and Autumn - the first chronicle in the history of Chinese literature. Fueled by his father's inspiration, Sima Qian started to compile Shiji in 109 BC. In 105 BC, Sima was among the scholars chosen to reform the calendar. As a senior imperial official, Sima was also in the position to offer counsel to the emperor on general affairs of state.

In 99 BC, Sima Qian became embroiled in the Li Ling Affair: Li Ling and Li Guangli , two military officers who led a campaign against the Xiongnu in the north, were defeated and taken captive. Emperor Han Wudi attributed the defeat to Li Ling, and all the officials in the government condemned Li Ling for the defeat. Sima was the only person to defend Li Ling, who had never been his friend but whom he respected. Emperor Han Wudi interpreted Sima’s defence of Li Ling as an attack on his brother-in-law, who had also fought against the Xiongnu without much success, and sentenced Sima to death. At that time, execution could be commuted either by money or castration. Since Sima did not have enough money to atone his "crime", he chose the latter and was then thrown into prison, where he endured three years. He described his pain thus: "When you see the jailer you abjectly touch the ground with your forehead. At the mere sight of his underlings you are seized with terror... Such ignominy can never be wiped away."

In 96 BC, on his release from prison, Sima chose to live on as a palace eunuch so as to complete his histories, rather than commit suicide as was expected of a gentleman-scholar. As Sima Qian's words explained: