Frazier – also known as Smokin’ Joe – had been receiving hospice care in Philadelphia after being diagnosed with cancer several weeks ago.
The 67-year-old was the first man to beat Muhammad Ali in 1971, but lost his next two bouts with Ali.
He held the world title between 1970 and 1973.
Frazier won an Olympic gold medal in 1964 after going to the Games as a replacement for Buster Mathis, who had beaten him in the trials but could not attend the Games due to an injury.
He won the heavyweight title in 1970, after Ali had been stripped of the championship in 1967 for refusing to fight in the Vietnam War, by defeating Jimmy Ellis in New York.
Three years later he lost his crown to George Foreman.
But the boxer is perhaps most widely-known for three great fights with Ali, including the epic “Thrilla in Manila” in 1975.
The pair had a fraught relationship dating back to taunts Ali directed at his rival in the build-up to their famous bouts.
But in recent years, they were reported to be on better terms.
“The world has lost a great champion. I will always remember Joe with respect and admiration,” said Ali, 69.
“My sympathy goes out to his family and loved ones.”
Frazier retired in 1976 after again being beaten by Foreman. He then made an unsuccessful comeback in 1981, fighting only once before ending his career for good.
“Goodnight Joe Frazier. I love you dear friend,” said Foreman on Twitter.
British former heavyweight champion Lennox Lewis told BBC Radio 5 live: “Without him, other boxing heroes wouldn’t be great because they tested their talent against him.
“He definitely was legendary and he made a great contribution to boxing.
“I’m so sad for his family. Nobody likes to hear about great heroes passing on. It’s a very sad day for boxing.” – BBC