Hi Yoni, welcome to the forum.
I can't speak to your profile as I'm not a doctor who makes these determinations. However, if you get an 82, I don't believe you will be eligible for the sayerot. Keep in mind that you can often negotiate a higher profile for yourself. You have to convince someone who can change your profile that it ought to be higher than what you got at lishkat hagiyus. If colorblindness automatically disqualifies you from having a 97 and it's a hard-and-fast rule, though, it may not be possible. But with an 82 you can serve in the infantry units and, I believe, eventually become an officer, if that's what you want to do.
As for Hebrew, if you can carry on a conversation, you do not need to go to ulpan. Keep working on it, of course, but it sounds like you're already in pretty good shape!
Mahal Nahal March '04 draft