Union leaders told the newspaper they are expecting absences to increase as part of a coordinated protest.
Many of the IRS employees who were called back are using a union contract provision that allows them to be absent from if they suffer a "hardship" during a federal government shutdown, labor groups told the Post.
"They are definitely angry that they are not getting paid, and maybe some of them are angry enough to express their anger this way," said Tony Reardon, president of the National Treasury Employees Union. "But these employees live paycheck to paycheck, and they can not scrape up the dollars to get to work or pay for child care."
The National Treasury Employees Union could not provide the Post with an official number of absent workers and did not immediately respond to CNN's request for comment.
IRS spokesman Matt Leas told the post that the agency is continuing its work to prepare for the start of tax-filing season, which begins on January 28.
"We are continuing our recall operations, and we continue to assess the situation at this time," Leas told the newspaper.