Supreme Court Justice John Dixon awarded her a record $4.5 million, including $3.9 million in economic damages relating to a loss of income.
But the Court of Appeal ruled that Wilson had not proven the company was responsible for her losing the film roles.
The articles alleged Wilson had lied about her age, her name and her upbringing in Australia, and were featured across Australian magazines, including Bauer's Australian Women's Weekly, New Weekly and OK Magazine.
The magazine publisher appealed against what was the largest defamation win in Australian legal history, arguing the size of the settlement set a unsafe precedent and there were errors of law in the judgment.
It found that the previous judge had relied on evidence from Wilson and two Hollywood agents to draw the conclusion that she had lost job opportunities.
"There was no basis in the evidence for making any award of damages for economic loss".
Speaking on behalf of Bauer Media, general counsel, Adrian Goss, said: "It was important for us to revisit the award of damages".
Rebel was not present in court as she is filming in Europe, but did tweet ahead of the appeal, pointing out that the verdict itself was not being questioned, only the amount of money paid.
Wilson said previously that she would donate the money made from the case to charity and use it to support the Australian film industry.
That sum would have been the largest defamation payment ever ordered by an Australian court.
Wilson said in April that she and her legal team were "very confident" the original defamation payout would be upheld.