Quick Read
- CRA spent $18 million on ‘Charlie’ chatbot, which had a low accuracy rate according to the Auditor General.
- The agency is hiring 1,700 call center workers to handle increased tax season demand.
- Charlie was upgraded to a generative AI model in November 2024, with claimed accuracy improvements.
- CRA call centers now answer up to 70% of unique calls, peaking at 92% response rates during the improvement plan.
- Government-wide cost-cutting measures contrast with CRA’s expansion and tech investments.
CRA’s $18 Million Chatbot Under Fire for Poor Accuracy
In the age of artificial intelligence, governments are eager to modernize public services. But for the Canada Revenue Agency (CRA), the journey hasn’t exactly been smooth sailing. The agency’s flagship digital assistant, ‘Charlie,’ a tax information chatbot, has become a lightning rod for criticism after a scathing Auditor General report revealed it provided incorrect answers to users most of the time.
Launched in February 2020 with high hopes, Charlie was meant to help Canadians navigate the country’s labyrinthine tax code. Yet, five years on, its performance has left many unimpressed. According to Auditor General Karen Hogan’s October 2024 report, Charlie answered correctly just two out of six sample questions—an accuracy rate of 33%, while other public web-based AI tools managed five out of six. This finding not only shocked members of parliament but also raised broader questions about how taxpayer money is being spent on digital solutions.
“Charlie’s responses tended to be brief, offering limited context and minimal additional information,” Hogan’s report noted. The chatbot’s failure to deliver reliable information is especially glaring given the $18 million invested since 2018, with the majority of funds allocated to salaries ($13.67 million) and IT consulting ($3.21 million). As Conservative critic Gérard Deltell bluntly put it, “We have to recognize that 33 per cent, it’s not satisfactory at all.”
Digital Transformation: Upgrades and Persistent Challenges
In response to mounting criticism, the CRA upgraded Charlie to a generative AI model in November 2024, aiming to boost accuracy and expand the range of questions it could answer. Preliminary internal tests suggested the new version could achieve around 90% accuracy, particularly on complex business-related queries. However, the agency admitted that a full accuracy assessment would require a comprehensive review of all chatbot interactions—a process not yet completed. The uncertainty lingers, leaving both the public and parliamentarians waiting for definitive results.
Charlie’s usage statistics are a testament to Canadians’ appetite for digital self-service. Since its inception, the chatbot has fielded more than seven million conversations and answered over 18 million questions. In the 2024 tax season alone, it handled hundreds of thousands of queries on personal taxes, business income, benefits, credits, and even mundane topics like updating personal information or contacting the agency. Interestingly, nearly 180,000 interactions were simply categorized as ‘chit chat,’ revealing the varied ways people engage with government technology.
Despite these impressive numbers, the struggle for accuracy remains a sore spot. The CRA’s own ‘accuracy threshold’ hovered around 70% for much of the tool’s history, meaning nearly one in three responses was wrong. While the agency touts improvements, skepticism persists—especially as the cost of developing AI tools continues to fall in the private sector, prompting questions about whether the government’s hefty investment was justified (National Post).
Massive Call Center Expansion Amid Service Delays
While Charlie grabs headlines, the CRA is also making waves with its ambitious plan to hire or rehire 1,700 call center agents for the 2025 tax season. The goal: ramp up staff to nearly 4,500 to handle the anticipated surge in taxpayer inquiries, particularly in the run-up to the April 30 filing deadline (SSBCrack News, el-balad.com).
During peak periods, CRA call centers receive over 300,000 calls daily—a staggering workload that has strained existing resources. Last year, the agency employed about 3,300 call center workers at the season’s peak. This year, it’s aiming for more than 1,000 additional hires. Melanie Serjak, CRA assistant commissioner, emphasized that such seasonal staffing increases are routine, allowing for flexibility and workload management throughout the year.
Behind this hiring spree lies a broader context: many federal departments are facing job cuts as the government looks to trim $60 billion from program spending and administrative costs over the next five years. Natural Resources Canada, the Public Service Commission, and the Department of Finance have all issued layoff notices to hundreds of employees. Yet, CRA officials, including assistant commissioner Maxime Guenette, have reassured staff that no immediate cuts are planned for the agency.
Service Improvements and Ongoing Scrutiny
Faced with criticism over long wait times and inaccurate information, the CRA was tasked by Finance Minister François-Philippe Champagne with a ‘100-day service improvement plan.’ By December 2024, the agency reported that its call centers’ response rates jumped from 35% to a targeted 70%, with peaks reaching as high as 92%. While that’s a dramatic improvement, the agency concedes that a perfect 100% response rate is unrealistic—even with expanded staffing and digital enhancements.
The accuracy of information provided to callers remains contentious. An Auditor General report found that only 17% of answers to individual tax questions were correct during a four-month audit, and just over 54% for business or benefit-related inquiries. The CRA disputes these findings, claiming its own internal reviews showed a 92% accuracy rate across more than 100,000 call evaluations this year. This tug-of-war over the numbers fuels ongoing debate about the effectiveness of CRA’s customer service, both online and by phone.
Looking ahead, the agency is developing a comprehensive three-to-five-year strategy to further improve service quality and operational efficiency. Wayne Long, Secretary of State for the CRA, highlighted the need for a long-term approach as the 100-day plan wraps up. The agency’s efforts to expand digital self-service options and enhance the generative AI chatbot are part of this broader push toward modernization.
Public Accountability and the Future of Government Tech
The CRA’s dual challenge—justifying the cost and performance of its digital tools while also meeting demand for human customer service—reflects a wider dilemma facing public agencies in the digital age. As private sector technology advances rapidly and AI solutions become more affordable, government projects like Charlie are increasingly scrutinized for cost-effectiveness and results.
The stakes are high. With millions of Canadians relying on the CRA for critical tax information and support, even minor improvements in accuracy and responsiveness can have significant ripple effects. Yet, the agency’s experience with Charlie and its call centers suggests that digital transformation is rarely a straight path. It requires ongoing investment, robust oversight, and a willingness to learn from missteps.
The CRA’s journey underscores the importance of transparency and accountability when deploying taxpayer-funded technology. While the push for modernization is essential, the agency’s experience with Charlie reveals that innovation must be matched by rigorous testing, clear benchmarks, and continuous feedback. As Canadians prepare for the 2025 tax season, the CRA’s efforts to balance human and digital service will remain under close watch—and serve as a litmus test for the future of government tech.