I am representing a Markov chain as a nested data structure, in Python as a dict of dicts of dicts... E.g. to understand what I mean, given the sentence 'this is purely an example, this is not serious.', I generate all the consecutive pairs and record the token that follows them and their frequencies:
{',': {'this': {'is': 1}},
'an': {'example': {',': 1}},
'example': {',': {'this': 1}},
'is': {'not': {'serious': 1}, 'purely': {'an': 1}},
'not': {'serious': {'.': 1}},
'purely': {'an': {'example': 1}},
'this': {'is': {'not': 1, 'purely': 1}}}
Then, I can query it using repeated item access. E.g. I can see that after 'this is' there's 'not' or 'purely', both with frequency 1.
In this contrived example the chain has a state size of 2, but I generate them with states of 3, 4, 5, 6. The text corpus is also huge, and the result is that the dictionary representing the chain takes tens of GB of RAM.
I was investigating alternative ways to store the Markov chain on disk. I've considered Neo4J, but it does not appear very well suited for this specific use case. The same applies to Postgres' ltree structure.
I've then settled on a simple table in a relational database, like the following (state size 4):
CREATE TABLE chain (
w1 varchar(20),
w2 varchar(20),
w3 varchar(20),
w4 varchar(20),
children json,
PRIMARY KEY(w1, w2, w3, w4)
);
There's a performance tradeoff when constructing the structure, but since it's only paid once it's acceptable.
Are there better way to store big Markov chains on disk, which allows querying without needed huge amounts of RAM?
A Markov Process is in a sense a probabilistic state machine, which satisfies the Markov property (that you can start the state machine from any state so that the past events should not affect the probabilities).
So, you should store a state index, by which you will query, and a Blob or something more descriptive which includes the states to which you can transition to and their probabilities.
When building the state index, you should not use just incremental index, but instead some kind of binary-search-like method, which makes sense in the domain of your machine learning application.
For example, you could have states 1000 1100 0100 and 0000 for "is", "not", "purely" and "this" (I am leaving out ",", "an", "example" for simplicity). Then, the state "this is", would be 0001, the first 00 denoting "this" and the second 01 denoting "is". Here I am assuming, that "this is" will contain full state e.g. that there will not be another "this is" in your data set. If that would be the case, I believe that would be a breach of Markov Property or flaw in your query logic (instead of bigrams you should be querying something else).
Anyway, this should be RAM efficient and could enable you many kinds of search strategies.