The paintings were completed last week by notable artist Mic Porter as part of the Creative Graffiti Pilot Program, established by local councils to mitigate graffiti in the area.
Its removal will be headed by Port Phillip Council.
Liberal MP David Southwick stated: "The first look at this mural, for me for and a number of constituents that have contacted me, is that it is highly offensive.”
"It certainly reminds people horrifically of the horrors of the past and imagery like that that was used during Nazi Germany," he continued. "And I think, particularly when you've got Melbourne's largest Jewish community shopping … the last thing that they would want to do when they enter a supermarket is to see that kind of imagery on display."
Port Phillip Mayor Heather Cunsolo said the council had "received a large volume of community complaints regarding some portraits that form part of this series.”
"We realise that regardless of the artist's intentions, the portraits have deeply upset and divided members of our community and for that we apologise," she said in a statement on Friday afternoon.
"When Council was first made aware that the artist's figures could be interpreted as Anti-Semitic, we reached out to several Jewish community leaders for advice.
"Whilst no concerns were raised, the current conflict has understandably heightened sensitivities and Council has no desire to add to the pain and distress many of our community are already feeling."
Porter’s work often features exaggerated and/or distorted facial features. He was selected to create the mural, Port Phillip Council said on their profile, "for his aesthetic and style but also because of the respect his experience commands throughout the graffiti scene.”
Unfortunately, anti-semitic caricatures have long featured Jewish people with enlarged noses.
The executive director of the Zionism Victoria lobby group Zeddy Lawrence stated: "Having looked at a number of the artist's works online, I can appreciate that this may be his style.
"If that's the case, it's just incredibly unfortunate that his comically grotesque images which are redolent of monstrous anti-Semitic caricatures appear as street art in such a notable Jewish neighbourhood.
"Someone, somewhere may have made a very poor judgement call as to what would be appropriate to feature on the facades of Carlisle Street."
Councillor Marcus Pearl wrote to the council requesting for it to be taken down and calling it “a governance failure” on their part.
He said in a statement: "I sincerely apologise for the distress caused by these artworks to our community and such in that such incidences of divisiveness are unacceptable at this time.”
The removal will begin Friday afternoon and is expected to be completed in a few days.