How would you implement a light weight, yet scalable, full text search index in Javascript that would only load indexes via static files accessible over http?
Note, I'm not looking for a Javascript search engine, like Lunr.js, that loads a single index file, since this doesn't scale.
I have about 100k documents I want to make searchable online, but I have very little money to spend on hosting a full text search engine (e.g. Elasticsearch). The documents can't necessarily be public, so indexing via Google and other public search engine isn't viable, and of course a paid private Google search server is also cost prohibitive. However, I can cheaply host a ton of JSON and other simple text files. So I'd like to implement a very rudimentary search engine using that. I'm not looking to implement a complex query language. Just simple keyword searches.
My naive approach is to parse all documents and create a bag-of-words for each file. Then, for each unique word, generate an index file representing a key/value store, listing the ID of every document that uses that word, along with a count of how many times the word occurs in the document.
Then, I can implement a simple Javascript or Python search by taking a user's search query, iterating over each search term, retrieving the index file for each term, and finding the document ID that has the highest number of counts for each term.
Something like:
def get_search_results(user_query, limit=10):
results = {} # {doc_id: score}
for term in user_query:
index_data = retrieve_index(term)
for doc_id, term_count in index_data.items():
results[doc_id] += term_count
results = sorted(results.items(), lambda o: o[1], reverse=True)[:limit]
return results
Obviously, this is a very naive approach and isn't terribly efficient. While it's very inexpensive to host, even for my relatively small number of documents, each index file is enormous and takes awhile for the script to parse through and aggregate.
However, I have to believe someone has encountered this problem before. Yet I can't find anything when searching for this type of client-side search engine using server side static index files. The only solutions I find are:
Neither of these solutions are cost effective.
Is there a better way I can structure my index files, or is there an existing tool or approach that does this type of search more efficiently?
Use an SQL.js fork with a Virtual Filesystem that supports Range requests so it efficiently only reads filesystem pages on demand. See the links below.
I also have a large catalog that I like to make searchable without the use of a query server (web3 project). Currently I'm using https://github.com/rhashimoto/wa-sqlite (with a custom Virtual Filesystem that supports any server that supports Range requests) to host the large sqlite file on Sia Skynet.
I'm still interested in a possible plaintext solution where only the index-to-the-index is served to the client in the most efficient way possible. Given enough static hosting space this must be possible. It will probably however not be worth the time to build considering SQL.js + HTTP VFS is already so reasonably efficient with some good indexes.
Honorable mentions that could have a HTTP VFS built in: