I have a large json file (around 16Gb) with the following structure:
{
"Job": {
"Keys": {
"JobID": "test123",
"DeviceID": "TEST01"
},
"Props": {
"FileType": "Measurements",
"InstrumentDescriptions": [
{
"InstrumentID": "1723007",
"InstrumentType": "Actual1",
"Name": "U",
"DataType": "Double",
"Units": "degC"
},
{
"InstrumentID": "2424009",
"InstrumentType": "Actual2",
"Name": "VG03",
"DataType": "Double",
"Units": "Pa"
}
]
},
"Steps": [
{
"Keys": {
"StepID": "START",
"StepResult": "NormalEnd"
},
"InstrumentData": [
{
"Keys": {
"InstrumentID": "1723007"
},
"Measurements": [
{
"DateTime": "2021-11-16 21:18:37.000",
"Value": 540
},
{
"DateTime": "2021-11-16 21:18:37.100",
"Value": 539
},
{
"DateTime": "2021-11-16 21:18:37.200",
"Value": 540
},
{
"DateTime": "2021-11-16 21:18:37.300",
"Value": 540
},
{
"DateTime": "2021-11-16 21:18:37.400",
"Value": 540
},
{
"DateTime": "2021-11-16 21:18:37.500",
"Value": 540
},
{
"DateTime": "2021-11-16 21:18:37.600",
"Value": 540
},
{
"DateTime": "2021-11-16 21:18:37.700",
"Value": 538
},
{
"DateTime": "2021-11-16 21:18:37.800",
"Value": 540
}
]
},
{
"Keys": {
"InstrumentID": "2424009"
},
"Measurements": [
{
"DateTime": "2021-11-16 21:18:37.000",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.100",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.200",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.300",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.400",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.500",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.600",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.700",
"Value": 1333.22
},
{
"DateTime": "2021-11-16 21:18:37.800",
"Value": 1333.22
}
]
}
]
}
]
}
}
The problem
I would like to split this file into multiple files by splitting the array "InstrumentData" because this array will be holding the major chunk of the data. Splitting this file into smaller files would enable me to parse the file without getting an out of memory exception.
Current State
public static void SplitJson(string filename, string arrayPropertyName)
{
string templateFileName = @"C:\Temp\template.json";
string arrayFileName = @"C:\Temp\array.json";
CreateEmptyFile(templateFileName);
CreateEmptyFile(arrayFileName);
using (Stream stream = File.OpenRead(filename))
using (JsonReader reader = new JsonTextReader(new StreamReader(stream)))
using (JsonWriter templateWriter = new JsonTextWriter(new StreamWriter(templateFileName)))
using (JsonWriter arrayWriter = new JsonTextWriter(new StreamWriter(arrayFileName)))
{
if (reader.Read() && reader.TokenType == JsonToken.StartObject)
{
templateWriter.WriteStartObject();
while (reader.Read() && reader.TokenType != JsonToken.EndObject)
{
string propertyName = (string)reader.Value;
reader.Read();
templateWriter.WritePropertyName(propertyName);
if (propertyName == arrayPropertyName)
{
arrayWriter.WriteToken(reader);
templateWriter.WriteStartObject(); // empty placeholder object
templateWriter.WriteEndObject();
}
else if (reader.TokenType == JsonToken.StartObject ||
reader.TokenType == JsonToken.StartArray)
{
templateWriter.WriteToken(reader);
}
else
{
templateWriter.WriteValue(reader.Value);
}
}
templateWriter.WriteEndObject();
}
}
// Now read the huge array file and combine each item in the array
// with the template to make new files
JObject template = JObject.Parse(File.ReadAllText(templateFileName));
using (JsonReader arrayReader = new JsonTextReader(new StreamReader(arrayFileName)))
{
int counter = 0;
while (arrayReader.Read())
{
if (arrayReader.TokenType == JsonToken.StartObject)
{
counter++;
JObject item = JObject.Load(arrayReader);
template[arrayPropertyName] = item;
string fileName = string.Format(@"C:\Temp\output_{0}_{1}_{2}.json",
template["name"], template["age"], counter);
File.WriteAllText(fileName, template.ToString());
}
}
}
// Clean up temporary files
File.Delete(templateFileName);
File.Delete(arrayFileName);
}
I am using this method to try and split the file into smaller files. However, this method can only split the files based on properties which are in the root level.
The question
Am I in the right track to tackle this problem? Is this an efficient way to tackle this? How do I split the JSON into multiple files by splitting the array in an efficient way? The JSON file should be split in a way that there is one file for each of the element in "InstrumentData" array. All the other properties and structures should be retained in the splitted files.
This is a little ugly solution and absolutely does not pretend to be the best but it allows to split a json with a similar structure of several GB in size into smaller files containing individual array members in a line-by-line manner. It preserves original redundant indentation and finalizing commas but these can be additionally fixed if necessary.
using var inputStream = File.OpenText("./input.json");
// Searching for the beginning of the array by the "InstrumentData" key
string? line;
while ((line = inputStream.ReadLine()) != null)
{
if (line.Contains("\"InstrumentData\""))
break;
}
// End of file reached, exiting
if (line == null)
return;
// Reading and splitting the "InstrumentData" array.
StreamWriter? outputFile = null;
var outputFilesCounter = 0;
var arrayLevel = 0;
var objectLevel = 0;
try
{
while ((line = inputStream.ReadLine()) != null)
{
// Track the levels of nesting within the array
arrayLevel += line.Count(c => c == '[');
objectLevel += line.Count(c => c == '{');
// Write a line into the currently opened output file stream
if (objectLevel > 0)
{
outputFile ??= File.CreateText($"./output_{outputFilesCounter++}.json");
outputFile.WriteLine(line);
}
arrayLevel -= line.Count(c => c == ']');
objectLevel -= line.Count(c => c == '}');
// End of an array member, flush the file stream
if (objectLevel == 0)
{
outputFile?.Dispose();
outputFile = null;
}
// End of array reached, exiting
if (arrayLevel < 0)
{
outputFile?.Dispose();
return;
}
}
}
finally
{
outputFile?.Dispose();
}
Here is one another approach you can use to split your large Json file using Cinchoo ETL - an open source library
Assumes the input json file comes with Job.Steps[*].InstrumentData[*] nodes, which requires 2 level of parsing to split the files by InstrumentData
First, break the input file by each Steps[*] node (aka. StepsFiles), then take each StepsFiles and break them by each InstrumentData[*] node.
Due to complexity of the code, drafted sample fiddle for review.
Sample fiddle: https://dotnetfiddle.net/j3Y03m
My solution here: https://dotnetfiddle.net/CufM4w
The idea is to process the document in two rounds using a relatively simple state automata.
Extract the root and every step templates (prefix and suffix), but skip the InstrumentData entirely.
On the contrary skip everything except InstrumentData and use the parts from #1 to build an output.
So the main code will look like (check the link above for the full source):
var reader = new StringReader(JsonText);
var splitter = DataSplitter.Create(reader);
reader = new StringReader(JsonText);
splitter.Split(reader, (step, index, content) => {
Console.WriteLine("=== step: {0}, index: {1} ===", step, index);
Console.WriteLine(content);
});
Hope it helped.