Thirty-eight white men allegedly from a far-right group assembled in the Grampians National Park over the long weekend, The Age reported on Thursday.

They allegedly chanted "white power", could be seen with Nazi tattoos, and addressed a Halls Gap local with a Sieg Heil.

Mr Andrews on Thursday said he recognised anti-Semitism was on the rise in Australia and overseas, despite being "evil and wicked".

"There's no place for that kind of anti-Semitism in our state," he told reporters on Thursday.

"There's no place for that of bigotry and hatred; there's no place for violence."

Mr Andrews said the Jewish community had made a profound contribution to Melbourne and Victoria.

"Anti-Semitic actions (and) activities affect the Jewish community deeply and it offends all of us," he said.

This comes after a man wearing a T-shirt representing a far-right group, known as Proud Boys, was removed by police from Melbourne's Invasion Day rally on Tuesday after he was involved in a brief scuffle with protesters.

"F*** off fascist, f*** off," the crowd yelled as he was led away.

Others were seen wearing Proud Boys clothing and draped in Australian flags as about 100 people marched down St Kilda Road in a "freedom" rally later that day.

Proud Boys was earlier this week labelled a "terrorist entity" by Canada's parliament.

Victoria Police confirmed the rally antagoniser was one of two men briefly detained for breaching the peace before being released without charge.

Australian Associated Press