Go's UTF-8 Support: An Interesting Limitation

I've been exploring Go's —a writing system where each consonant-vowel sequence is written as a unit. In Unicode, this often involves combining a base consonant character with one or more combining marks that represent vowels or other modifiers.

For example:

• The Tamil letter க (U+0B95) represents the consonant sound "ka"




• To represent "ki" you'd combine க with the vowel sign ி (U+0BBF), resulting in கி.




• The vowel sign ி is a combining mark, specifically classified as a "Non-Spacing Mark" in Unicode.

Here's where the problem arises.

Go's language specification allows Unicode letters in identifiers but excludes combining marks. Specifically, identifiers can include characters that are classified as "Letter" (categories Lu, Ll, Lt, Lm, Lo, or Nl) and digits, but not combining marks (categories Mn, Mc, Me).

Examples of Combining Marks in Tamil

Let's look at how Tamil characters are formed:

• Standalone Consonant: க (U+0B95) - Allowed in Go identifiers.




• Consonant + Vowel Sign: கா (U+0B95 U+0BBE) - Not allowed because ா (U+0BBE) is a combining mark (Mc).




• Consonant + Vowel Sign: கி (U+0B95 U+0BBF) - Not allowed because ி (U+0BBF) is a combining mark (Mn).




• Consonant + Vowel Sign: கூ (U+0B95 U+0BC2) - Not allowed because ூ (U+0BC2) is a combining mark (Mc).

In the identifier எண்ணிக்கை ("number"), the characters include combining marks:

• எ (U+0B8E) - Letter, allowed.




• ண் (U+0BA3 U+0BCD) - Formed by ண (U+0BA3) and the virama ் (U+0BCD), a combining mark (Mn).




• ண (U+0BA3) - Letter, allowed.




• ிக்கை - Contains combining marks like ி (U+0BBF) and ை (U+0BC8).

Because these combining marks are not allowed in Go identifiers, the compiler throws errors when it encounters them.

Why Chinese Characters Work but Tamil Doesn't

Chinese characters are generally classified under the "Letter, Other" (Lo) category in Unicode. They are standalone symbols that don't require combining marks to form complete characters. This is why identifiers like 消息 work perfectly in Go.

Practical Implications

The inability to use combining marks in identifiers has significant implications for scripts like Tamil:

• Limited Expressiveness: Without combining marks, it's nearly impossible to write meaningful identifiers in Tamil.




• Educational Barriers: Using native scripts can make learning to code more accessible, but these limitations hinder that possibility, particular for languages that follow abugida-based writing system.




• Inclusivity Challenges: While Go aims for inclusivity with its UTF-8 support, the restrictions on combining marks exclude many languages that rely on them.

Wrapping Up

Go's UTF-8 support is a great step toward making programming more inclusive. However, the exclusion of combining marks in identifiers creates barriers for languages like Tamil, Hindi and Arabic, where combining marks are integral to the script.

As a developer from Tamilnadu, working primarily in Go, this discovery was both exciting and a bit disappointing. It highlights the complexities of true internationalization in programming languages.

Who codes in native languages for building Software Products !?!!!!

Absolutely! Not so much apart from the East-asian region, where ‘abugida’ based writing system isn’t followed.

And, obviously, creators of Go would not have intended the UTF-8 compliance for ‘native-language coding’ at first place. The reason was more towards providing better ASCII processing, alignment with modern web standards, consistent string handling and a step towards interoperability.

This attempt was just my curiosity in understanding how far we can take the UTF-8 Compliance in Go. As someone who works on building scalable, distributed fintech systems in Go, I found it essential to be aware of these nuances.

Thats about it. Thanks for reading along.

Happy coding :) May the code be with you.