Acquis 8 - Transcoding

The string.transcode function converts strings between text encodings and binary-to-text formats. It enables interoperability with web APIs, legacy systems, and binary protocols.

Basic usage

Transcode a string from one encoding to another:

local decoded = string.transcode("SGVsbG8gV29ybGQ=", "base64", "utf-8")
assert(decoded == "Hello World")

local encoded = string.transcode("Hello World", "utf-8", "base64")
assert(encoded == "SGVsbG8gV29ybGQ=")

The function signature is string.transcode(data, from, to [, ignorebad]).

Supported encodings

Character encodings

These encodings represent text as sequences of characters:

-- ASCII: 7-bit characters (0-127)
string.transcode("Hello", "ascii", "utf-8")

-- UTF-8: variable-width Unicode
string.transcode("日本語", "utf-8", "utf-16le")

-- UTF-8 with BOM: adds/strips byte order mark
string.transcode("Hello", "utf-8", "utf-8bom")  -- prepends EF BB BF
string.transcode("\xEF\xBB\xBFHello", "utf-8bom", "utf-8")  -- strips BOM

-- UTF-16LE: little-endian UTF-16
string.transcode("A", "utf-8", "utf-16le")  -- "A\0"

-- ISO-8859-1 (Latin-1): Western European
string.transcode("café", "utf-8", "iso-8859-1")  -- "caf\xE9"
string.transcode("café", "utf-8", "latin-1")     -- alias

Binary-to-text encodings

These encodings represent arbitrary bytes as printable text:

-- Base64
string.transcode("user:password", "utf-8", "base64")  -- "dXNlcjpwYXNzd29yZA=="
string.transcode("dXNlcjpwYXNzd29yZA==", "base64", "utf-8")  -- "user:password"

-- URL encoding
string.transcode("hello world", "utf-8", "url")  -- "hello%20world"
string.transcode("a%3D1%26b%3D2", "url", "utf-8")  -- "a=1&b=2"

-- Hexadecimal
string.transcode("Lus", "utf-8", "hex")  -- "4c7573"
string.transcode("4c7573", "hex", "utf-8")  -- "Lus"

Character set conversion

Convert between character encodings via Unicode:

-- UTF-8 to UTF-16LE
local utf16 = string.transcode("Hello 日本", "utf-8", "utf-16le")
-- H\0e\0l\0l\0o\0 \0\xE5\x65\x2C\x67

-- UTF-16LE back to UTF-8
local utf8 = string.transcode(utf16, "utf-16le", "utf-8")
assert(utf8 == "Hello 日本")

Supplementary plane characters (emoji, rare CJK) use surrogate pairs in UTF-16LE:

-- 𝌆 (U+1D306) encoded as surrogate pair
local tetragram = string.transcode("𝌆", "utf-8", "utf-16le")
assert(tetragram == "\x34\xD8\x06\xDF")  -- D834 DF06 in little-endian

Binary format chaining

Convert directly between binary-to-text formats:

-- Base64 to hex
local hex = string.transcode("SGVsbG8=", "base64", "hex")
assert(hex == "48656c6c6f")  -- "Hello" in hex

-- Hex to URL encoding
local url = string.transcode("48656c6c6f", "hex", "url")
assert(url == "Hello")  -- printable chars unchanged

Error handling

Invalid input throws an error:

-- Invalid base64
local ok, err = catch string.transcode("!!invalid!!", "base64", "utf-8")
assert(not ok)
assert(err:find("invalid base64"))

-- Character not representable in target encoding
local ok, err = catch string.transcode("日本語", "utf-8", "ascii")
assert(not ok)
assert(err:find("cannot be encoded as ASCII"))

-- Malformed UTF-8
local ok, err = catch string.transcode("\xFF\xFE", "utf-8", "hex")
assert(not ok)
assert(err:find("invalid UTF-8"))

Graceful degradation with ignorebad

Pass true as the fourth argument to skip invalid characters:

-- Skip non-ASCII characters
local result = string.transcode("Hello 日本", "utf-8", "ascii", true)
assert(result == "Hello ")

-- Skip non-Latin1 characters
local result = string.transcode("€100", "utf-8", "iso-8859-1", true)
assert(result == "100")  -- € (U+20AC) not in Latin-1

-- Skip malformed input
local result = string.transcode("valid\xFF\xFEtext", "utf-8", "hex", true)
assert(result == "76616c6964text")  -- skips bad bytes

Real-world examples

HTTP Basic Authentication

local function makeBasicAuth(username, password)
    local credentials = username .. ":" .. password
    local encoded = string.transcode(credentials, "utf-8", "base64")
    return "Basic " .. encoded
end

local header = makeBasicAuth("admin", "secret123")
-- "Basic YWRtaW46c2VjcmV0MTIz"

URL query string encoding

local function encodeQuery(params)
    local parts = {}
    for key, value in pairs(params) do
        local k = string.transcode(key, "utf-8", "url")
        local v = string.transcode(value, "utf-8", "url")
        parts[#parts + 1] = k .. "=" .. v
    end
    return table.concat(parts, "&")
end

local query = encodeQuery({name = "José", city = "São Paulo"})
-- "name=Jos%C3%A9&city=S%C3%A3o%20Paulo"

Hex dump utility

local function hexdump(data)
    local hex = string.transcode(data, "utf-8", "hex")
    local lines = {}
    for i = 1, #hex, 32 do
        lines[#lines + 1] = string.sub(hex, i, i + 31)
    end
    return table.concat(lines, "\n")
end

print(hexdump("Hello, World!"))
-- 48656c6c6f2c20576f726c6421

Reading files with different encodings

local function readLatin1File(path)
    local f = io.open(path, "rb")
    local content = f:read("*a")
    f:close()
    return string.transcode(content, "iso-8859-1", "utf-8")
end

local text = readLatin1File("legacy_document.txt")

Motivation

Lus strings are raw byte buffers with no inherent encoding. While flexible, this creates challenges when working with external systems that expect specific text representations.

Web API integration

Base64 and URL encoding are ubiquitous:

-- Without transcode: manual encoding
local b64_chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/"
-- ... 50+ lines of encoding logic

-- With transcode: one function call
local encoded = string.transcode(data, "utf-8", "base64")

Legacy system compatibility

Many legacy systems use Latin-1 or other single-byte encodings:

-- Convert modern UTF-8 to legacy encoding
local legacy = string.transcode(modern_text, "utf-8", "iso-8859-1")

Cross-platform text handling

UTF-16LE is common on Windows:

-- Read Windows clipboard or registry data
local utf8_text = string.transcode(windows_data, "utf-16le", "utf-8")

The unified string.transcode API handles all these cases without external dependencies.