Character encodings
Find the character encoding of a file or of bytes (byte order mark, UTF-8, then statistics for Western, Central European, Asian and Cyrillic encodings) with a confidence and the other candidates, and convert a file to another encoding without ever leaving it half written. Typical uses: load a CSV exported in windows-1252 by an old program, check a file before importing it, write a file in the encoding a partner expects.
file detect_encodingfile detect_encodingstring detect_encodingstring detect_encodingfile convert_encodingfile convert_encoding
file detect_encoding <path>
Description
The encoding of a file: JSON {file, size, bytes_read, encoding, confidence, method, bom, ascii_only, valid_utf8, candidates: [{encoding, confidence}], same_text_as, line_breaks}
The encoding of a file: JSON {file, size, bytes_read, encoding, confidence, method, bom, ascii_only, valid_utf8, candidates: [{encoding, confidence}], same_text_as, line_breaks}
Parameters
path: The path of the file - String - required
path: The path of the file - String - required
admin
csv write "tmp/doc_webdata_clients.csv" "[{\"nom\": \"Dupont\", \"prenom\": \"Hélène\", \"ville\": \"Saint-Denis\", \"montant\": \"12,50 €\"}, {\"nom\": \"Payet\", \"prenom\": \"Jérôme\", \"ville\": \"Le Tampon\", \"montant\": \"7,00 €\"}, {\"nom\": \"Hoarau\", \"prenom\": \"Cécile\", \"ville\": \"Saint-Pierre\", \"montant\": \"3,20 €\"}]" "{\"separator\": \";\", \"encoding\": \"windows-1252\", \"line_end\": \"crlf\"}"; file detect_encoding "tmp/doc_webdata_clients.csv"
mentdb
{"file": "tmp/doc_webdata_clients.csv", "size": 127, "bytes_read": 127, "encoding": "windows-1252", "confidence": 0.98, "method": "statistics", "bom": false, "ascii_only": false, "valid_utf8": false, "candidates": [{"encoding": "windows-1252", "confidence": 0.98}, {"encoding": "windows-1250", "confidence": 0.98}, {"encoding": "windows-1257", "confidence": 0.735}, {"encoding": "IBM850", "confidence": 0.735}, {"encoding": "windows-1254", "confidence": 0.368}, {"encoding": "x-MacRoman", "confidence": 0.368}], "same_text_as": ["windows-1254", "windows-1256"], "line_breaks": "CRLF"}
file detect_encoding <path> <json_params>
Description
The encoding of a file with parameters: max_bytes (8 MB: the bytes read, 64 MB at most), candidates (["windows-1252", "IBM850"]: the statistics choose among these encodings only)
The encoding of a file with parameters: max_bytes (8 MB: the bytes read, 64 MB at most), candidates (["windows-1252", "IBM850"]: the statistics choose among these encodings only)
Parameters
path: The path of the file - String - required
json_params: The parameters, as a JSON object - String - required
path: The path of the file - String - required
json_params: The parameters, as a JSON object - String - required
admin
file detect_encoding "tmp/doc_webdata_clients.csv" "{\"candidates\": [\"windows-1252\", \"IBM850\", \"x-MacRoman\"]}"
mentdb
{"file": "tmp/doc_webdata_clients.csv", "size": 127, "bytes_read": 127, "encoding": "windows-1252", "confidence": 0.98, "method": "statistics", "bom": false, "ascii_only": false, "valid_utf8": false, "candidates": [{"encoding": "windows-1252", "confidence": 0.98}, {"encoding": "IBM850", "confidence": 0.735}, {"encoding": "x-MacRoman", "confidence": 0.368}], "same_text_as": [], "line_breaks": "CRLF"}
Used in use cases (1 recipe)
Integrator & ETL developer › Detect the encoding of a legacy export and convert it to UTF-8 before loading it
Integrator & ETL developer › Detect the encoding of a legacy export and convert it to UTF-8 before loading it
string detect_encoding <data>
Description
The encoding of bytes given in base64: the same JSON as file detect_encoding, without file and size
The encoding of bytes given in base64: the same JSON as file detect_encoding, without file and size
Parameters
data: The bytes, in base64 - String - required
data: The bytes, in base64 - String - required
admin
string detect_encoding "UHJpeCA6IDEyIKQgbGUga2lsbywgY711ciBkZSBwYWxtaWVy"
mentdb
{"bytes_read": 36, "encoding": "ISO-8859-15", "confidence": 0.73, "method": "statistics", "bom": false, "ascii_only": false, "valid_utf8": false, "candidates": [{"encoding": "ISO-8859-15", "confidence": 0.73}, {"encoding": "windows-1252", "confidence": 0.365}, {"encoding": "ISO-8859-1", "confidence": 0.365}, {"encoding": "windows-1250", "confidence": 0.365}, {"encoding": "ISO-8859-2", "confidence": 0.365}, {"encoding": "windows-1254", "confidence": 0.365}], "same_text_as": [], "line_breaks": "none"}
A DOS text (code page 850)
admin
string detect_encoding "UoJmgnJlbmNlO0xpYmVsbII7UXVhbnRpdIIKQS0xMjtDbIIghSBtb2xldHRlOzMKQi0wNztUb3VybmV2aXMgY3J1Y2lmb3JtZTsxMgo="
mentdb
{"bytes_read": 77, "encoding": "IBM850", "confidence": 0.98, "method": "statistics", "bom": false, "ascii_only": false, "valid_utf8": false, "candidates": [{"encoding": "IBM850", "confidence": 0.98}, {"encoding": "IBM437", "confidence": 0.98}, {"encoding": "windows-1252", "confidence": 0.653}, {"encoding": "windows-1250", "confidence": 0.653}, {"encoding": "windows-1254", "confidence": 0.653}, {"encoding": "windows-1257", "confidence": 0.653}], "same_text_as": ["IBM437"], "line_breaks": "LF"}
Japanese
admin
string detect_encoding "k/qWe4zqgsyDZYNMg1iDZ4LFgreBQpW2jpqDUoFbg2iCzJS7kuiCzINlg1iDZ4LFgreBQg=="
mentdb
{"bytes_read": 52, "encoding": "Shift_JIS", "confidence": 0.99, "method": "statistics", "bom": false, "ascii_only": false, "valid_utf8": false, "candidates": [{"encoding": "Shift_JIS", "confidence": 0.99}, {"encoding": "windows-31j", "confidence": 0.9}, {"encoding": "IBM437", "confidence": 0.078}, {"encoding": "IBM850", "confidence": 0.067}, {"encoding": "x-MacRoman", "confidence": 0.037}, {"encoding": "windows-1256", "confidence": 0.019}], "same_text_as": [], "line_breaks": "none"}
string detect_encoding <data> <json_params>
Description
The encoding of bytes with parameters: input (base64, hex), candidates
The encoding of bytes with parameters: input (base64, hex), candidates
Parameters
data: The bytes - String - required
json_params: The parameters, as a JSON object - String - required
data: The bytes - String - required
json_params: The parameters, as a JSON object - String - required
admin
string detect_encoding "436166e9206372e86d65" "{\"input\": \"hex\"}"
mentdb
{"bytes_read": 10, "encoding": "windows-1252", "confidence": 0.73, "method": "statistics", "bom": false, "ascii_only": false, "valid_utf8": false, "candidates": [{"encoding": "windows-1252", "confidence": 0.73}, {"encoding": "ISO-8859-15", "confidence": 0.73}, {"encoding": "ISO-8859-1", "confidence": 0.73}, {"encoding": "windows-1250", "confidence": 0.73}, {"encoding": "ISO-8859-2", "confidence": 0.73}, {"encoding": "windows-1257", "confidence": 0.365}], "same_text_as": ["ISO-8859-15", "ISO-8859-1", "windows-1256", "windows-1254", "ISO-8859-9"], "line_breaks": "none"}
file convert_encoding <source> <target> <from> <to>
Description
Convert a file: JSON {source, target, from, detected, to, bom_removed, bom_written, bytes_in, bytes_out, characters, lines, invalid, unmappable, first_invalid, first_unmappable}. from can be auto (detected; an error below the confidence 0.5)
Convert a file: JSON {source, target, from, detected, to, bom_removed, bom_written, bytes_in, bytes_out, characters, lines, invalid, unmappable, first_invalid, first_unmappable}. from can be auto (detected; an error below the confidence 0.5)
Parameters
source: The path of the file to read - String - required
target: The path of the file to write (can be the source) - String - required
from: The encoding of the source, or auto - String - required
to: The encoding of the target - String - required
source: The path of the file to read - String - required
target: The path of the file to write (can be the source) - String - required
from: The encoding of the source, or auto - String - required
to: The encoding of the target - String - required
admin
file convert_encoding "tmp/doc_webdata_clients.csv" "tmp/doc_webdata_clients_utf8.csv" "auto" "UTF-8"
mentdb
{"source": "tmp/doc_webdata_clients.csv", "target": "tmp/doc_webdata_clients_utf8.csv", "from": "windows-1252", "detected": true, "confidence": 0.98, "to": "UTF-8", "bom_removed": false, "bom_written": false, "bytes_in": 127, "bytes_out": 138, "characters": 127, "lines": 4, "invalid": 0, "unmappable": 0, "first_invalid": null, "first_unmappable": null}
The euro sign does not exist in ISO-8859-1
admin
file convert_encoding "tmp/doc_webdata_clients_utf8.csv" "tmp/doc_webdata_clients_latin1.csv" "UTF-8" "ISO-8859-1"
mentdb
ERROR: Sorry, an error appears (the character U+20AC (€) at line 2, column 33 cannot be written in ISO-8859-1 (set unmappable to replace or skip)).
file convert_encoding <source> <target> <from> <to> <json_params>
Description
Convert a file with parameters: unmappable (error, replace, skip: a character the target cannot write), replacement (the text written for it, ? by default), invalid (error, replace, skip: invalid bytes of the source, replaced by U+FFFD), bom (false: true writes a byte order mark in UTF-8, UTF-16LE/BE, UTF-32LE/BE), line_breaks (keep, lf, crlf), min_confidence (0.5, with auto), overwrite (true)
Convert a file with parameters: unmappable (error, replace, skip: a character the target cannot write), replacement (the text written for it, ? by default), invalid (error, replace, skip: invalid bytes of the source, replaced by U+FFFD), bom (false: true writes a byte order mark in UTF-8, UTF-16LE/BE, UTF-32LE/BE), line_breaks (keep, lf, crlf), min_confidence (0.5, with auto), overwrite (true)
Parameters
source: The path of the file to read - String - required
target: The path of the file to write - String - required
from: The encoding of the source, or auto - String - required
to: The encoding of the target - String - required
json_params: The parameters, as a JSON object - String - required
source: The path of the file to read - String - required
target: The path of the file to write - String - required
from: The encoding of the source, or auto - String - required
to: The encoding of the target - String - required
json_params: The parameters, as a JSON object - String - required
admin
file convert_encoding "tmp/doc_webdata_clients_utf8.csv" "tmp/doc_webdata_clients_latin1.csv" "UTF-8" "ISO-8859-1" "{\"unmappable\": \"replace\", \"replacement\": \"EUR\"}"
mentdb
{"source": "tmp/doc_webdata_clients_utf8.csv", "target": "tmp/doc_webdata_clients_latin1.csv", "from": "UTF-8", "detected": false, "to": "ISO-8859-1", "bom_removed": false, "bom_written": false, "bytes_in": 138, "bytes_out": 133, "characters": 127, "lines": 4, "invalid": 0, "unmappable": 3, "first_invalid": null, "first_unmappable": "U+20AC (€) at line 2, column 33"}
UTF-8 with a byte order mark and Unix line breaks, in place
admin
file convert_encoding "tmp/doc_webdata_clients_utf8.csv" "tmp/doc_webdata_clients_utf8.csv" "UTF-8" "UTF-8" "{\"bom\": true, \"line_breaks\": \"lf\"}"
mentdb
{"source": "tmp/doc_webdata_clients_utf8.csv", "target": "tmp/doc_webdata_clients_utf8.csv", "from": "UTF-8", "detected": false, "to": "UTF-8", "bom_removed": false, "bom_written": true, "bytes_in": 138, "bytes_out": 137, "characters": 127, "lines": 4, "invalid": 0, "unmappable": 0, "first_invalid": null, "first_unmappable": null}
Used in use cases (1 recipe)
Integrator & ETL developer › Detect the encoding of a legacy export and convert it to UTF-8 before loading it
Integrator & ETL developer › Detect the encoding of a legacy export and convert it to UTF-8 before loading it
