Character Encoding

Overview

Users sometimes encounter problems with FTP transfers that garble non-English characters in filenames, such as umlauts, accented letters or completely different scripts like Chinese or Arabic.

FTP is a rather old protocol and things we take for granted now were not even considered when it was designed. One of these things is support for non-English characters in filenames. When the FTP protocol was designed, computers mostly spoke English and were unable to display any non-English characters. As such, the FTP protocol was designed to be used with English characters only, namely 7-bit ASCII.

The problem is that many FTP clients and servers purposely violate the FTP specifications in order to support other, non-standard character sets. Which of these character sets are used is not subject to any negotiation. For any character set in existence, you can find a server using it with no way of detecting the proper encoding. The result: non-English characters are not transferred correctly.

To solve this problem, the FTP protocol has been extended in a backwards compatible way to use UTF-8 as the character set. (This solution is backwards compatible only with servers in compliance with the original specifications.)

If you have problems with filenames containing any foreign characters, this can have two reasons:

• The server or client follows the original specifications by the letter and rightfully rejects those filenames
• The server or client violates the specifications and uses a custom encoding that the other party does not understand

Both FileZilla Client and Server are fully compliant with the updated specifications and use UTF-8. FileZilla will not break FTP specifications by supporting non-standard encodings in order to accommodate the user.

If you have problems with other clients or servers, please upgrade (or ask the server to upgrade) to FTP software capable of UTF-8 or refrain from using foreign characters. Anything else is in violation of the FTP specifications and will only work if you manually ensure that the server and client use the same character encoding (which may not even be possible).

Technical details

The FTP protocol is specified in RFC 959, which was published in 1985. The FTP protocol is designed on top of the original Telnet protocol, which is specified in RFC 854. The relevant sections of the Telnet specification regarding FTP are those covering the Network Virtual Terminal (NVT). According to RFC 854, the NVT requires the use of (7-bit) ASCII as the character set. Use of any other character set requires explicit negotiation. This character set only contains 127 different characters: English letters and numbers, punctuation characters and a few control characters. Accented letters, umlauts or other scripts are not contained in the ASCII character set.

In order to support non-English characters, the FTP specifications were extended in 1999 in RFC 2640. This extension requires the use of UTF-8 as the character set. This character set is a strict superset of ASCII, every valid ASCII character is also the same character in UTF-8. The UTF-8 character set can display any valid Unicode character. That includes umlauts, accented letters and also different scripts. This extension is fully backwards compatible with RFC 959.

As long as you're not using any non-English characters, it doesn't matter if the software you are using supports RFC 2640 or not. However, if you use non-English characters without using RFC 2640 compatible software, there will be problems--problems which are entirely self-made by not obeying the specifications.

UTF8 feature negotiation

An RFC 2640 compliant server must support the FEAT command and must include a line containing UTF8 in its response:

Command:  FEAT
Response: 211-Features:
 [...]
Response:  UTF8
 [...]
Response: 211 End

　( 2011.04.13 20:30 )  : ;[HLMasterServer/127.0.0.1:27015];sample CS severr;info=\protocol\47\challenge\%u\players\max\24\bots\gamedir\cstrike\map\de_nuke\type\d\password\os\w\secure\lan\version\1.1.2.5\region\255\product\cstrike;[SourceMasterServer/127.0.0.1:27015];sample L4D2 severr;info=\protocol\7\challenge\%u\players\max\4\bots\gamedir\left4dead2\map\c2m4_barns\password\os\w\lan\region\255\gameport\27015\specport\dedicated\1\gametype\coop,empty\appid\550\type\d\secure\1\version\2.0.1.1\product\left4dead2;gamedata=\protocol\7\challenge\%u\region\255\gamedata\g:l4d2,coop,realism,survival,versus,teamversus,scavenge,teamscavenge,key:2011\gdversion\1这些是MasterServer处理HL1和HL2引擎游戏服务器的字段信息