What is +ACY- doing in the data?
For a full list of BASHing data blog posts see the index page. Almost all the data files I audit are in UTF-8, but the files have often started out in other encodings. This can lead to some hilarious mojibake and loads of fun for me as I try to reverse the encoding conversion failures. Last week a file appeared with mojibake I'd never seen before. Here are the original characters followed by the character strings as they appeared in the audit file:   --> +AME-   --> +AOE-  & --> +ACY-   --> +AOk-   --> +AO0-   --> +APM   --> +APY-   --> +APw-  After some googling I found the culprit, namely RFC 2152. This 1997 recommendation (not a standard) defines "UTF-7" and is now regarded as obsolete. UTF-7 was designed to be an email-safe way to handle Unicode characters by converting them into 7-bit US ASCII strings. To show how UTF-7 works, I'll encode the Latin capital letter a with acute, "", Unicode U+00C1. In UTF-16, U+00C1 is 0000 0000 1100 0001  Starting from the left, group the UTF-16 binary in lots of 6 bits:  000000 001100 0001  If the last lot has fewer than 6 bits, pad it out with trailing zeroes:  000000 001100 000100  Read each 6-bit lot as if it was in Base64: A M E  Add a leading "+" to indicate that the block is  UTF-16-modified-then-Base64-encoded-and-tweaked  UTF-16-modified-then-Base64-encoded-and-tweaked Add a trailing "-" to indicate the end of the block  To decode a small number of UTF-7 strings, I recommend the excellent string-functions.com Web page with a UTF-7-vs-US-ASCII table. To convert a block of text riddled with UTF-7 constructions, the best bet is iconv on the command line:  A UTF-7 code block is an example of a reversible replo: it's easily fixed by changing the encoding of the file. Last update: 2021-07-14  The blog posts on this website are licensed under a  Creative Commons Attribution-NonCommercial 4.0 International License