Found this off another site and i had the same problem on my 1995 merc 115.
Does your alarm go off as soon as you turn the key?
I was fishing a tourney on fork with a good friend and i turned the key over and the alarm went off and would not go off, so we trolled and fished around the marina, went to weigh in and talked to a mechanic buddy of my friend and the first thing he said is that the float is stuck.
Took it to Tommy at Rockwall Marine and sure enough get a call from Tommy and it was the float.
Had to update the oil tank, the 95 model was to old to find an exact match. never had a problem with alarms again unless it was actually low.
Give Tommy a call or send him an e-mail he is great at helping ppl out, been doing business with him for 3 years and won't take either of my boats anywhere else.
http://www.fishingworld.com/vDealers/TX/RockwallMarine/ (972) 771-4442