Don't use the translation table (Localizer) use the manual translation for the voc.
I mean, take this mode:
"Per language terms. Different terms will be allowed for each language and they can be translated."
set it Language-independet.
All terms must be there in every language, then translate it by:
www.yoursite.com/en/admin/content/taxonomy/1/translation

and don't synchonize the Taxonomy Terms - then it will work! (I've tried some hours to get that figured out)

I only don't get the Voc.Name translated this way.

I hope it helps