Victorians are being fined by police for activities they are allowed to do under public health orders, including going to the supermarket and exercising, while others have been fined for not being at home because they were in hotel quarantine, community legal services say.

A lawyer with Inner Melbourne Community Legal, Lloyd Murphy, is concerned increased powers given to police to issue fines during the Covid-19 pandemic are being used inappropriately – and that people are struggling to contest fines due to language barriers or fear.

The premier, Daniel Andrews, said in July that one in four people who had tested positive for Covid-19 were not at home when door-knocked by authorities, that 100 cases had been referred to Victoria police, and that it was “unacceptable” that people with Covid-19 were not home.

But Murphy said just 42 fines had been issued to people for not staying at home while infected with or being a close contact of someone with coronavirus.

Of the 42 people who had been fined, several had approached Inner Melbourne Community Legal for assistance, Murphy said.

“We have found that some of them were in fact in hotel quarantine and that’s why they were not at their family residence,” he said. “We also have stories of people in the locked-down community housing towers arranging their own hotel accommodation to better isolate from and protect their families, and then police send infringement notices to their homes. In other cases, the Department of Health and Human Services put them in hotels for the safety of the other tower residents, yet police fined them when they were not home.”

Murphy said the service was yet to successfully dispute any of the fines, despite earlier assurances by police that common sense and “discretion” would be used when reviewing them.

Murphy said the legal service was also assisting international students with poor English who were stopped by police while exercising, or while on their way to the supermarket, but who were unable to explain themselves due to a language barrier. He said most were reluctant to allow the legal service to help them dispute the fine through the courts because they were worried it might jeopardise their visa. Other cases being examined by the centre include people stopped by police for jaywalking, but who later received a fine in the mail for breaching Covid-19 restrictions.

“The government and police have done an amazing job at highlighting a daily reel of ‘idiots’ who got a Covid fine in the previous 24 hours, and as a result it has helped influence public opinion and the mainstream media to support these police powers and to justify them,” Murphy said. “But they’re not highlighting the hundreds of others who got a fine for not being able to communicate with police, or who were wrongly fined when they were allowed outside.”

Concerns have also been raised by the Independent Broad-based Anti-corruption Commission (Ibac) deputy commissioner, Katie Miller, that Covid-19 may increase the risk of police misconduct.

“The breadth of the Covid-19 restrictions and the speed at which they were introduced, along with subsequent changes, understandably creates some uncertainty about how they are applied to any given activity or circumstance,” Miller wrote in her latest newsletter.

“The public health emergency presented by Covid-19 may increase the risk of rationalising misconduct on the basis that ‘the ends justifies the means’.”

One woman who lives in Fitzroy North told Guardian Australia she was left distressed after two police officers approached her and noted down her details after she took her 11-month-old to Edinburgh Gardens to crawl on the grass. She still does not know if she will receive a fine in the mail as police did not tell her if she was breaking the law.

“We had a combative five to 10-minute interaction where one of the officers was unclear whether or not this was within regulations and said he needed to investigate,” she said. “He insisted on taking my details and repeatedly said he needed them because ‘the baby could crawl at home in the lounge room’ and ‘the exercise regulation is for things like running which you can’t do at home’. I repeatedly asked if I was doing something to break regulations because, if so, I would stop, but he said that he wasn’t clear and so would investigate to find out. Two other people saw the whole thing and stopped to ask if I was OK.”

A Victoria police spokeswoman said police were “not in a position” to detail whether the woman was in breach of public health directions. The department of health said if a young child cannot be left at home they can accompany an exercising parent.

Ariel Couchman is the chief executive of Youthlaw, Victoria’s free legal service for people under 25, which is currently helping 21 young people with Covid fines. These included a 16-year-old refugee living in Australia with his extended family who was walking with a friend when approached by a stranger asking for directions. Police fined the teenager and his friend $1,652 each for breaking the two-person exercise rule while they were giving directions.

Youthlaw is also assisting a young man with mental health issues, including schizophrenia, whose finances are managed by state trustees. He was fined for being outside riding a bike. Meanwhile, a young woman with a developmental delay was fined for shopping more than 5km from her home, despite it being her nearest shopping centre. Police did not attempt to find out why she was there, Couchman said.