Business
Jobs
  • About Us
  • Solutions
    • Job Postings
      Post your job and receive qualified candidates in 48h.
    • Candidate Assessments
      500+ technical and psychological tests, plus anti-fraud.
    • Headhunting
      Tailor-made executive search from start to finish.
    • Payroll + EOR
      Payroll dispersal and EOR across 15+ LATAM countries.
  • Pricing
  • Jobs

0

88
Views
Implementing scalable text search using static index files accessible from the web?

How would you implement a light weight, yet scalable, full text search index in Javascript that would only load indexes via static files accessible over http?

Note, I'm not looking for a Javascript search engine, like Lunr.js, that loads a single index file, since this doesn't scale.

I have about 100k documents I want to make searchable online, but I have very little money to spend on hosting a full text search engine (e.g. Elasticsearch). The documents can't necessarily be public, so indexing via Google and other public search engine isn't viable, and of course a paid private Google search server is also cost prohibitive. However, I can cheaply host a ton of JSON and other simple text files. So I'd like to implement a very rudimentary search engine using that. I'm not looking to implement a complex query language. Just simple keyword searches.

My naive approach is to parse all documents and create a bag-of-words for each file. Then, for each unique word, generate an index file representing a key/value store, listing the ID of every document that uses that word, along with a count of how many times the word occurs in the document.

Then, I can implement a simple Javascript or Python search by taking a user's search query, iterating over each search term, retrieving the index file for each term, and finding the document ID that has the highest number of counts for each term.

Something like:

def get_search_results(user_query, limit=10):
    results = {} # {doc_id: score}
    for term in user_query:
        index_data = retrieve_index(term)
        for doc_id, term_count in index_data.items():
            results[doc_id] += term_count
    results = sorted(results.items(), lambda o: o[1], reverse=True)[:limit]
    return results

Obviously, this is a very naive approach and isn't terribly efficient. While it's very inexpensive to host, even for my relatively small number of documents, each index file is enormous and takes awhile for the script to parse through and aggregate.

However, I have to believe someone has encountered this problem before. Yet I can't find anything when searching for this type of client-side search engine using server side static index files. The only solutions I find are:

  1. Buy or rent a big expensive full text search server and query that from the client.
  2. Generate the full text search index on the server side, then load that massive index on the client (e.g. Lunr.js)

Neither of these solutions are cost effective.

Is there a better way I can structure my index files, or is there an existing tool or approach that does this type of search more efficiently?

about 4 years ago · Juan Pablo Isaza
1 answers
Answer question

0

Use an SQL.js fork with a Virtual Filesystem that supports Range requests so it efficiently only reads filesystem pages on demand. See the links below.

I also have a large catalog that I like to make searchable without the use of a query server (web3 project). Currently I'm using https://github.com/rhashimoto/wa-sqlite (with a custom Virtual Filesystem that supports any server that supports Range requests) to host the large sqlite file on Sia Skynet.

I'm still interested in a possible plaintext solution where only the index-to-the-index is served to the client in the most efficient way possible. Given enough static hosting space this must be possible. It will probably however not be worth the time to build considering SQL.js + HTTP VFS is already so reasonably efficient with some good indexes.

  • use SQlite over torrent: https://boredcaveman.xyz/post/0x2_static-torrent-website-p2p-queries.html (based on wa-sqlite, lazy partial loading index)
  • use SQLite over HTTP using range requests: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ (lazy loading indexes)
  • https://github.com/riyaz-ali/sqlite3.js (lazy partial loading indexes)
  • https://github.com/ydylla/wa-sqlite/tree/webtorrent (lazy loading index over torrent)

Honorable mentions that could have a HTTP VFS built in:

  • client side FTS: https://github.com/tinysearch/tinysearch (no lazy loading index)
  • client-side FTS: https://stork-search.net/ (no lazy loading index yet)
about 4 years ago · Juan Pablo Isaza Report
Answer question
Find remote jobs

Discover the new way to find a job!

Top jobs
Top job categories
Business
Post vacancy Pricing Sales
Legal
Terms and conditions Privacy policy
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Show me some job opportunities
There's an error!