Empresas
Empleos
  • Sobre nosotros
  • Soluciones
    • Publicación de vacantes
      Publica tu vacante y recibe candidatos calificados en 48h.
    • Evaluación de candidatos
      500+ pruebas técnicas y psicológicas, más anti-fraude.
    • Headhunting
      Búsqueda ejecutiva a la medida de principio a fin.
    • Nómina + EOR
      Dispersión de nómina y EOR en más de 15 países de LATAM.
  • Precios
  • Empleos

0

91
Vistas
Implementing scalable text search using static index files accessible from the web?

How would you implement a light weight, yet scalable, full text search index in Javascript that would only load indexes via static files accessible over http?

Note, I'm not looking for a Javascript search engine, like Lunr.js, that loads a single index file, since this doesn't scale.

I have about 100k documents I want to make searchable online, but I have very little money to spend on hosting a full text search engine (e.g. Elasticsearch). The documents can't necessarily be public, so indexing via Google and other public search engine isn't viable, and of course a paid private Google search server is also cost prohibitive. However, I can cheaply host a ton of JSON and other simple text files. So I'd like to implement a very rudimentary search engine using that. I'm not looking to implement a complex query language. Just simple keyword searches.

My naive approach is to parse all documents and create a bag-of-words for each file. Then, for each unique word, generate an index file representing a key/value store, listing the ID of every document that uses that word, along with a count of how many times the word occurs in the document.

Then, I can implement a simple Javascript or Python search by taking a user's search query, iterating over each search term, retrieving the index file for each term, and finding the document ID that has the highest number of counts for each term.

Something like:

def get_search_results(user_query, limit=10):
    results = {} # {doc_id: score}
    for term in user_query:
        index_data = retrieve_index(term)
        for doc_id, term_count in index_data.items():
            results[doc_id] += term_count
    results = sorted(results.items(), lambda o: o[1], reverse=True)[:limit]
    return results

Obviously, this is a very naive approach and isn't terribly efficient. While it's very inexpensive to host, even for my relatively small number of documents, each index file is enormous and takes awhile for the script to parse through and aggregate.

However, I have to believe someone has encountered this problem before. Yet I can't find anything when searching for this type of client-side search engine using server side static index files. The only solutions I find are:

  1. Buy or rent a big expensive full text search server and query that from the client.
  2. Generate the full text search index on the server side, then load that massive index on the client (e.g. Lunr.js)

Neither of these solutions are cost effective.

Is there a better way I can structure my index files, or is there an existing tool or approach that does this type of search more efficiently?

about 4 years ago · Juan Pablo Isaza
1 Respuestas
Responde la pregunta

0

Use an SQL.js fork with a Virtual Filesystem that supports Range requests so it efficiently only reads filesystem pages on demand. See the links below.

I also have a large catalog that I like to make searchable without the use of a query server (web3 project). Currently I'm using https://github.com/rhashimoto/wa-sqlite (with a custom Virtual Filesystem that supports any server that supports Range requests) to host the large sqlite file on Sia Skynet.

I'm still interested in a possible plaintext solution where only the index-to-the-index is served to the client in the most efficient way possible. Given enough static hosting space this must be possible. It will probably however not be worth the time to build considering SQL.js + HTTP VFS is already so reasonably efficient with some good indexes.

  • use SQlite over torrent: https://boredcaveman.xyz/post/0x2_static-torrent-website-p2p-queries.html (based on wa-sqlite, lazy partial loading index)
  • use SQLite over HTTP using range requests: https://phiresky.github.io/blog/2021/hosting-sqlite-databases-on-github-pages/ (lazy loading indexes)
  • https://github.com/riyaz-ali/sqlite3.js (lazy partial loading indexes)
  • https://github.com/ydylla/wa-sqlite/tree/webtorrent (lazy loading index over torrent)

Honorable mentions that could have a HTTP VFS built in:

  • client side FTS: https://github.com/tinysearch/tinysearch (no lazy loading index)
  • client-side FTS: https://stork-search.net/ (no lazy loading index yet)
about 4 years ago · Juan Pablo Isaza Denunciar
Responde la pregunta
Encuentra empleos remotos

¡Descubre la nueva forma de encontrar empleo!

Top de empleos
Top categorías de empleo
Empresas
Publicar vacante Precios Comercial
Legal
Términos y condiciones Política de privacidad
© 2026 PeakU Inc. All Rights Reserved.
Andres GPT
Recomiéndame algunas ofertas
Necesito ayuda