Few people have taken advantage of relaxed rules offering early access to retirement savings for those facing financial hardship due to the coronavirus pandemic. Instead, most 401(k) plan participants appear to be staying the course and haven't made significant changes to their retirement savings.

Fidelity, the nation's largest 401(k) provider, said contributions to its workplace savings plans actually increased slightly in the third quarter, despite market swings and economic uncertainty. At Fidelity, the average 401(k) balance increased to $109,600 by the end of September, a 5% increase from the second quarter and up from 4% from a year ago.

Under the federal CARES Act, retirement savings plan participants who have been affected by the coronavirus have until the end of December to take up to $100,000 out of their 401(k), 403(b) or individual retirement account without paying a 10% early withdrawal penalty.

However, not many have done it.

More from Invest in You:
Spend instead of save? Here's why – and how you can change it
Amid coronavirus changes, some millennials and Gen Z stumble into saving
The ultimate retirement planning guide for 2020

About 1.3 million people — or 5.2% of participants — withdrew money from Fidelity workplace savings plans between April 1 and Oct. 31. At Vanguard, 4.5% of participants had made a 401(k) withdrawal under the CARES Act rules by the end of September. And 7% of eligible participants at T. Rowe Price have taken a coronavirus-related distribution from their 401(k).

Fidelity said the average 401(k) plan withdrawal was $10,000 — a relatively small amount, but one that could eventually grow to be a significant slice of retirement income if left untouched. If you're 35, a $10,000 nest egg could grow to nearly $100,000 by the time you're 70, assuming a 7% annual return, according to calculations by Bankrate.com.

That's why financial advisors warn against tapping your 401(k) and putting your future financial stability at risk.

"Even if it's possible to borrow from your 401(k) or take a distribution ... consider this a last resort," said Carrie Schwab-Pomerantz, a certified financial planner and president of the Charles Schwab Foundation. "While present circumstances may be difficult, I'd counsel anyone to avoid jeopardizing their future retirement unless absolutely necessary.

"You may not appreciate the full consequences until much later."

Also, financial advisors say workers should be mindful of the potential tax hit of taking money out of a 401(k) plan. While the early withdrawal penalty should be waived for a coronavirus-related distribution, the money that you withdraw from your 401(k) will be considered regular income and will be subject to federal taxes.

The IRS says the income can be spread over three years. Taxes you pay also could be refunded if you pay back the entire withdrawal amount within three years, but that will require filing an amended tax return.

"The bottom line: There are a number of ifs, ands [and] buts with this, not to mention the logistics of doing it all correctly per the IRS," said CFP Stacy Miller, a partner at Bright Investments, which has offices in Auburn, Alabama, and Tampa, Florida. "My fear is that these employees are getting no information and are just doing it anyway without understanding it all.

"They could end up with a tax bill they're not expecting."

SIGN UP: Money 101 is an 8-week learning course to financial freedom, delivered weekly to your inbox.

CHECK OUT: I have 9 side hustles and bring in up to $4,000 each month: Here's my best advice via Grow with Acorns+CNBC.

Disclosure: NBCUniversal and Comcast Ventures are investors in Acorns.