You need to have upgraded the mines once before you can remove them.
If you try to destroy them before the first upgrade, you keep an abandoned building.
If you set them up for upgrading without the materials needed, let builders dismantle them and try to cancel the upgrade then to trick the destruction, you'll keep a pit.

Basically, you'll need candles to upgrade any mine once, and after that upgrade is completed, then you'll be able to remove it once it's empty again.

As a side note, if you upgrade the mine a second time, it becomes limitless, so there's no real need to destroy mines unless you're not happy with their placement and want to move them.

Just read this, so is this also true of a small quarry? I tried removing it after it was at 0%resource and now I have a biggish pit in the middle of town. Thx

Yep, it's true of all the Mines/Quarries. We didn't want to make it too "easy" to get rid of them, without letting people stuck with things at 0%.
The 1st upgrade isn't too difficult to get and grants you another full mine/quarry that you can either upgrade a 2d time to become infinite or that you can remove as needed when it also runs out.

Ok so next time I need to upgrade once then I remove. Or upgrade a 2nd time for infinite quarry. Correct?

Correct.
(Make sure you let that 1st upgrade complete. Interrupting midway through the upgrade to try and take advantage of the builders' deconstructing the 1st level doesn't work ;) )