HMRC has been accused of breaching data protection laws by taking the "voice IDs" of over 5m people without their consent.
When users call HMRC, they are asked to repeat the phrase "my voice is my password" on an automated line as a security precaution before they can access services.
But data watchdog Big Brother Watch says the taxman is forcing people to be "railroaded into a mass ID scheme" as people are not given the choice to opt in or out, which they say is in breach of data protection laws.
FOI requests obtained by the group found that HMRC holds around 5.1m voiceprints, and has refused to disclose which other government departments the recordings have been shared with, or how they are stored and used, or if it's possible to delete a voice ID.
The Information Commissioner's Office (ICO) is now investigating this issue.
Voice IDs are designed so that only the individual can access their account using their own voice. But problems have been raised with the technology before. In 2017, a BBC reporter managed to hack into a blank HSBC bank account that had the security system in place.
Silkie Carlo, director of Big Brother Watch, said:
“Taxpayers are being railroaded into a mass ID scheme that is incredibly disturbing. The tax man is building Big Brother Britain by imposing biometric ID cards on the public by the back door.
The rapid growth of the British database state is alarming. These voice IDs could allow ordinary citizens to be identified by government agencies across other areas of their private lives. HMRC should delete the 5 million voiceprints theyve taken in this shady scheme, observe the law and show greater respect to the public.”
Pat Walshe, director of another group, Privacy Matters, added: "HMRCs voiceprint scheme appears to be almost surreptitious, failing to meet basic data protection principles.
"The non-transparent manner harvesting of peoples data and significant questions of lawfulness are troubling.
"Given the significant number of citizens involved, and the potential for broader use of biometric voiceprints by government agencies, the ICO could issue a notice requiring the temporary suspensions of the scheme."
A spokesperson for HMRC said: "Our Voice ID system is very popular with customers as it gives a quick and secure route into our systems. The Voice ID data storage meets the highest government and industry standards for security."