Owen Smith says he “stood by his principles” in calling for another EU referendum – a move which resulted in his sacking from Labour’s shadow cabinet.
The former shadow Northern Ireland secretary said Jeremy Corbyn had made a “mistake” in firing him.
He also said the party should “shift its position” on Brexit.
Mr Smith was asked to stand down on Friday after he wrote an article for the Guardian calling for a second vote.
Speaking about his sacking on BBC Radio 4’s Today programme, he said: “I think it is a mistake, for Jeremy Corbyn in particular, who has always understood the value of people standing by their principles.
“It is the position that he has often adopted, and it is certainly a value in him that others have extolled.”
“In truth I think that is all I have done. I have stood by my principles.”
Mr Smith said leaving the EU was “the biggest economic crisis that our country will have faced for many, many generations” and he believed Labour should stand against it.
Referring to Mr Corbyn’s views on Brexit as “a more Eurosceptic position”, he added: “It’s the first instance that I can think of in living memory of a government pursuing a policy that they know is going to make our economy smaller and reduce people’s livelihoods and life chances and I cannot understand why we in Labour would support that.”
Party figures have criticised Mr Corbyn’s decision to sack Mr Smith, with Labour peer Peter Hain describing the dismissal as a “Stalinist purge”.
The former Northern Ireland secretary said Mr Smith was widely respected for his work in the role.
Labour MP Chuka Umunna said it was “extraordinary” that the shadow minister had been sacked for advocating a Brexit policy which, he said, had wide support in the party.
Fellow MP Anna Turley said Mr Smith’s departure was “disappointing” and he would be a loss to the front bench, while former cabinet minister Ben Bradshaw told Mr Smith he was “very sorry” to see him go.
By Jonathan Blake, BBC political correspondent
In a phone call from Jeremy Corbyn and what a source described as a “civil conversation”, Owen Smith was told he was being sacked.
Mr Smith’s comments on Brexit policy were seemingly a step too far for the party leadership, who’d welcomed him back into the fold after his leadership challenge in 2016.
The angry response from some MPs has again laid bare their opposition to Jeremy Corbyn as leader.
But Mr Corbyn’s grip on power in the party has never been stronger.
The fact that he can make this move without fear of open rebellion, is a demonstration of that.
However shadow home secretary Diane Abbott said although Mr Smith was “a valued colleague”, he could not sit on the opposition front bench while advancing “a position which was simply not Labour Party policy”.
Ms Abbott, who in November told constituents she would push for a referendum on the final Brexit deal – before clarifying she wanted a Parliamentary vote – said Mr Smith would be able to “make a contribution to the debate” outside the shadow cabinet.
BBC Newsnight’s political editor Nick Watt said supporters of Mr Corbyn thought Mr Smith “wasn’t very collegiate” and had defied collective responsibility.
Mr Watt also reported that friends of Mr Smith complained that journalists knew about the sacking before Mr Corbyn phoned him – a claim disputed by the Labour leader’s office.
Mr Smith has been replaced by Rochdale MP and shadow housing minister Tony Lloyd.
Mr Corbyn said that Mr Lloyd was “highly experienced” and “committed to ensuring that peace in Northern Ireland is maintained”, as well as helping to steer the devolution deal “back on track”.
In a tweet, Mr Smith said his concerns about Brexit, which he outlined in a Guardian article, were shared by other supporters and members of the Labour party.
In the article, Mr Smith called for Labour to back membership of the EU single market.
The Labour leader announced last month that the party wanted the UK to be a permanent member of a customs union with the EU after Brexit.
But Mr Smith, who unsuccessfully challenged Mr Corbyn for the party leadership in 2016, insisted Labour needed to do more than “just back a soft Brexit or guarantee a soft border in Ireland”.
He wrote: “If we insist on leaving the EU then there is realistically only one way to honour our obligations under the Good Friday Agreement and that is to remain members of both the customs union and the single market.”
Labour’s shadow Brexit Secretary Sir Keir Starmer has said the party was “not calling for a referendum at this stage”.