Yo! The hottest pools in the large municipal swimming pool, next to the city campground, were too hot for me to touch! But much more humble and elegant in a way than touristy Blue Lagoon, and definitely full of friendly Icelanders who seem to go there almost every day.
I've driven the whole Ring Road, and there are great spots everywhere.
One of my favorites is Thingvellir, where you can see the actual Mid Atlantic Rift breaking apart, complete with pillow basalts showing beautiful flow structures.
Definitely get out of Reykjavik. The countryside is phenomenal.