Some water companies put some lime in the water to solve certain problems, but it is unusual. Most alkalinity is from natural sources and just isn't removed. If the alkalinity is due to calcium, the reverse osmosis treatment will likely work; for other things, maybe not. If you already have the equipment, you can give it a try and see what happens. Otherwise, I would use one of the commonly available ion exchange units (e.g Brita, and several others.
Call your city and ask them for a report of the city drinking water. I was just mailed one from the city I live in. It should tell you the hardness and perhaps the alkalinity of your cities water and we can take it from there.
When I needed this information for setting a dishwasher, years ago, I just phoned up a rather nice lady at the water company who could give me the information in a choice of 5 or 6 types of unit. I remember frantically flipping through the manual to find one I recognised
Just give them your post code (or Zip code) and they'll be able to tell you more than you need to know right off the page
"Flatter Me, and I May Not Believe You. Criticize Me, and I May Not like You. Ignore Me, and I May Not Forgive You. Encourage Me, and I Will Not Forget You."