Character encoding

Character encoding is the process of interpreting bytes to readable characters. UTF-8 is the dominant encoding since 2009 and is promoted as a de-facto standard [1].

UTF-8

Terminal

Most terminals use UTF-8 by default. However, some (e.g: gnome-terminal, rxvt-unicode) need to be launched from a UTF-8 locale. To do that, set the codeset part of your locale to .UTF-8 and reload your session. See Locale#Setting the locale for instructions.

xterm - Run with the argument -u8 or configure resource xterm*utf8: 2.

Unicode character insertion

See List of applications/Utilities#Text input.

LibreOffice includes a built-in charmap in Insert > Special Character showing character block, hex, and decimal encodings.
Compose Key: XCompose can be set up to insert special characters.
AltGr can be set up to access additional keyboard levels providing extra characters as well as Unicode control characters such as ZWNJ and RTL mark.
Vim: using Ctrl+v uXXXX in insert mode, where XXXX is the hexadecimal code point.

URL encoding

URIs accept US-ASCII characters only and use percent-encoding to encode non-ASCII characters. This can result in very long and human-unreadable URIs.

In Firefox, it is possible to copy decoded URLs by enabling the browser.urlbar.decodeURLsOnCopy flag in about:config, or by inserting a space to the start of the URL, then selecting it (with the space) and copying it. However, this trick does not work on Chromium, and there is no equivalent flag. Alternatively, select starting at the end of the URL until right after the https:// part, then copy.

For command line usage, you can use python to translate encoded URLs from stdin.

$ python3 -c "import sys; from urllib.parse import unquote; print(unquote(sys.stdin.read().strip()))"

Troubleshooting

Encoding problems are usually due to two programs communicating with different encodings, with one side typically not using UTF-8, resulting in mojibake.

Warning: It is highly recommended to set the codeset of your locale to .UTF-8. Otherwise, conversion from UTF-8 to non-Unicode encoding can result in a loss of information.

Incorrect archive encoding

On older versions of Windows (XP, Vista, and 7), File Explorer uses different encoding when creating a zip archive with certain locales. To extract properly, use unzip -O followed with the target encoding. E.g: CP936 is a common encoding in old versions of Windows.

$ unzip -O CP936 file.zip

If unsure about the needed charset, dry-run without extraction by adding the -l flag:

$ unzip -lO SJIS file.zip

Japanese versions of Windows encode ZIP archives with Shift-JIS. Use shift-jis:

$ unzip -O shift-jis nihongo.zip

Chinese versions of Windows encode ZIP archives with gbk:

$ unzip -O gbk file.zip

Alternatively, use unzip-natspec^AUR for auto-detecting the targeted encoding.

Incorrect file name encoding

Use convmv for encoding-conversion mv:

$ convmv -f SOURCE_ENCODING -t UTF-8 --nosmart file

By default, convmv shows what would be done without actual moving. After figuring out the original encoding using -f (e.g: for Chinese GBK), add the --notest option to proceed with the move operation.

By default, convmv skips file name conversion if it is already UTF8-encoded. Use the --nosmart option to force the conversion.

Use convmv --list to find the supported encodings.

Incorrect file encoding

Use the iconv command to convert the format. For example:

$ iconv -f SOURCE_ENCODING -t UTF-8 -o new-file origin-file

-f specifies the original encoding and -t specifies the output encoding. Use iconv -l to query all supported encodings and -o to specify the output file.

Vim

If the locale is UTF-8, opening other char-encoded files may be garbled. You can add a fallback adding to vimrc a line similar to:

set fileencodings=utf8,cp936,gb18030,big5

Alternatively, you can explicitly set it by :set fileencoding=ansi. Vim will do the conversion via iconv automatically. See :h charset-conversion.

Incorrect MP3 ID3 tag encoding

To modify the MP3 file tag, convert using python-mutagen or mp3unicode:

$ mid3iconv -e SOURCE_ENCODING file.mp3

If file modification is undesired, you can tweak the behavior of media players. For players that use GStreamer as the backend, such as Rhythmbox and totem, set the environment variable:

GST_ID3_TAG_ENCODING=GBK:UTF-8:GB18030

Quod Libet player supports tag editing and setting ID3v2 encoding. Go to File > Plugins > Advanced Preferences (or File > Preferences > Advanced for quodlibet-git^AUR), click I know what I'm doing and enter a space-separated list of encodings in the ID3 encodings field. You can also edit the configuration file manually:

~/.config/quodlibet/config

...
[editing]
id3encoding = gbk
...

Note: Quod Libet supports utf8 encoding by default.

Incorrect mount encoding

Generally, the mounted character set is different from the locales, which can be set by modifyinig fstab. If the locale is utf8, modify the line to:

/etc/fstab

...
/dev/sdxx /media/win ntfs defaults,iocharset=utf8 0 0

If the locale is GBK, it should be:

/etc/fstab

...
/dev/sdxx /media/win ntfs defaults,iocharset=cp936 0 0
...

Incorrect Samba encoding

When using Arch as a Samba server, adding the following line to /etc/samba/smb.conf can solve the garbled problem of Windows clients:

/etc/samba/smb.conf

...
unix charset=gb2312
...

Incorrect FTP encoding

If you use UTF8 locale, the downloaded file name from a non-Unicode-encoded server might be garbled. For lftp, make the following settings under .lftp/rc:

.lftp/rc

...
set ftp:charset "gbk"
set file:charset "UTF-8"
...

For gftp, you can do the following settings in .gftp/gftprc:

.gftp/gftprc

...
remote_charsets=gb2312
...

However, the downloaded file name is still garbled and needs to be patched and compiled. The patch address is: https://www.teatime.com.tw/%7Etommy/linux/gftp_remote_charsets.patch