JSON Lines
Read and write JSON Lines (NDJSON) files and texts, one JSON value per line. Typical uses: load log exports or machine learning data sets, prepare files for bulk APIs.
jsonl read <path>
Description
Read a file of JSON Lines: a JSON array of the values of all the lines (objects or not). The first bad line is an error
Read a file of JSON Lines: a JSON array of the values of all the lines (objects or not). The first bad line is an error
Parameters
path: The file - String - required
path: The file - String - required
admin
jsonl write "tmp/doc_api.jsonl" "[{\"id\": 1}, {\"id\": 2}]"; jsonl read "tmp/doc_api.jsonl"
mentdb
[{"id": 1}, {"id": 2}]
jsonl read <path> <json_params>
Description
Read with parameters: objects_only (false; true: a line that is not an object is bad), skip_bad (false: the first bad line is an error naming its number; true: the result is {rows, count, bad_count, bad}, with bad = [{line, error, text}]), max_bad (100 lines kept in bad), max_lines (1000000, 10000000 at most: beyond it is always an error), max_line_bytes (16 MB), max_bytes (of the result, 64 MB, 1 GB at most), offset (data lines to pass over without reading them), limit (stop after this number of values)
Read with parameters: objects_only (false; true: a line that is not an object is bad), skip_bad (false: the first bad line is an error naming its number; true: the result is {rows, count, bad_count, bad}, with bad = [{line, error, text}]), max_bad (100 lines kept in bad), max_lines (1000000, 10000000 at most: beyond it is always an error), max_line_bytes (16 MB), max_bytes (of the result, 64 MB, 1 GB at most), offset (data lines to pass over without reading them), limit (stop after this number of values)
Parameters
path: The file - String - required
json_params: The parameters - String - required
path: The file - String - required
json_params: The parameters - String - required
admin
jsonl read "tmp/doc_api.jsonl" "{\"offset\": 1, \"limit\": 1}"
mentdb
[{"id": 2}]
Keep the good lines and report the others
admin
jsonl write "tmp/doc_api2.jsonl" "[{\"id\": 1}, [1, 2]]"; jsonl read "tmp/doc_api2.jsonl" "{\"skip_bad\": true, \"objects_only\": true}"
mentdb
{"rows": [{"id": 1}], "count": 1, "bad_count": 1, "bad": [{"line": 2, "error": "a JSON object is expected", "text": "[1, 2]"}]}
Used in use cases (1 recipe)
Integrator & ETL developer › Import a JSON Lines export and report the bad lines
Integrator & ETL developer › Import a JSON Lines export and report the bad lines
jsonl write <path> <json_rows>
Description
Write a JSON array as a file, one compact value per line, each line ended by a line feed. The file is replaced all at once: a failure leaves the old file as it was. On Linux and macOS an existing file keeps its rights and its owner, and a symbolic link stays a link. A path that exists and is not a regular file (named pipe, device) is refused. JSON {path, lines, bytes, append}
Write a JSON array as a file, one compact value per line, each line ended by a line feed. The file is replaced all at once: a failure leaves the old file as it was. On Linux and macOS an existing file keeps its rights and its owner, and a symbolic link stays a link. A path that exists and is not a regular file (named pipe, device) is refused. JSON {path, lines, bytes, append}
Parameters
path: The file (its folder must exist) - String - required
json_rows: The JSON array - String - required
path: The file (its folder must exist) - String - required
json_rows: The JSON array - String - required
admin
jsonl write "tmp/doc_api.jsonl" "[{\"id\": 1}, {\"id\": 2}]"
mentdb
{"path": "tmp/doc_api.jsonl", "lines": 2, "bytes": 20, "append": false}
jsonl write <path> <json_rows> <json_params>
Description
Write with parameters: append (true: add at the end, with a line feed first if the last line had none)
Write with parameters: append (true: add at the end, with a line feed first if the last line had none)
Parameters
path: The file - String - required
json_rows: The JSON array - String - required
json_params: The parameters - String - required
path: The file - String - required
json_rows: The JSON array - String - required
json_params: The parameters - String - required
admin
jsonl write "tmp/doc_api.jsonl" "[{\"id\": 3}]" "{\"append\": true}"
mentdb
{"path": "tmp/doc_api.jsonl", "lines": 1, "bytes": 10, "append": true}
Used in use cases (1 recipe)
Integrator & ETL developer › Import a JSON Lines export and report the bad lines
Integrator & ETL developer › Import a JSON Lines export and report the bad lines
jsonl to_rows <text>
Description
Read a JSON Lines text as rows: a JSON array of objects. A line that is not an object is bad
Read a JSON Lines text as rows: a JSON array of objects. A line that is not an object is bad
Parameters
text: The text - String - required
text: The text - String - required
admin
jsonl to_rows "{\"id\": 1, \"name\": \"Ana\"} {\"id\": 2, \"name\": \"Bob\"} "
mentdb
[{"id": 1, "name": "Ana"}, {"id": 2, "name": "Bob"}]
A line in error
admin
jsonl to_rows "{\"id\": 1} not json"
mentdb
ERROR: Sorry, an error appears (line 2: invalid JSON at position 0 (unexpected character)).
jsonl to_rows <text> <json_params>
Description
The same with parameters: objects_only (true; false: any JSON value), skip_bad (false: the first bad line is an error naming its number; true: the result is {rows, count, bad_count, bad}, with bad = [{line, error, text}]), max_bad (100 lines kept in bad), max_lines (1000000, 10000000 at most: beyond it is always an error), max_line_bytes (16 MB), max_bytes (of the result, 64 MB, 1 GB at most), offset (data lines to pass over without reading them), limit (stop after this number of values)
The same with parameters: objects_only (true; false: any JSON value), skip_bad (false: the first bad line is an error naming its number; true: the result is {rows, count, bad_count, bad}, with bad = [{line, error, text}]), max_bad (100 lines kept in bad), max_lines (1000000, 10000000 at most: beyond it is always an error), max_line_bytes (16 MB), max_bytes (of the result, 64 MB, 1 GB at most), offset (data lines to pass over without reading them), limit (stop after this number of values)
Parameters
text: The text - String - required
json_params: The parameters - String - required
text: The text - String - required
json_params: The parameters - String - required
admin
jsonl to_rows "{\"id\": 1} not json [1, 2] {\"id\": 12345678901234567890} " "{\"skip_bad\": true}"
mentdb
{"rows": [{"id": 1}, {"id": 12345678901234567890}], "count": 2, "bad_count": 2, "bad": [{"line": 2, "error": "invalid JSON at position 0 (unexpected character)", "text": "not json"}, {"line": 3, "error": "a JSON object is expected", "text": "[1, 2]"}]}
The first line only
admin
jsonl to_rows "{\"id\": 1, \"name\": \"Ana\"} {\"id\": 2, \"name\": \"Bob\"} " "{\"limit\": 1}"
mentdb
[{"id": 1, "name": "Ana"}]
Used in use cases (1 recipe)
Integrator & ETL developer › Import a JSON Lines export and report the bad lines
Integrator & ETL developer › Import a JSON Lines export and report the bad lines
jsonl from_rows <json_rows>
Description
Write a JSON array as JSON Lines text, each line ended by a line feed. An empty array gives an empty text
Write a JSON array as JSON Lines text, each line ended by a line feed. An empty array gives an empty text
Parameters
json_rows: The JSON array - String - required
json_rows: The JSON array - String - required
admin
jsonl from_rows "[{\"id\": 1, \"price\": 1.50}, {\"id\": 2, \"price\": 12345678901234567890}]"
mentdb
{"id": 1, "price": 1.50} {"id": 2, "price": 12345678901234567890}
