CrazyEngineers
CrazyEngineers
  • I'm wondering what'd be faster. Let's say I've about 10K records which I need to search for autocomplete query. I believe I've two options-
    • Source the data from a plain file - either plain text or maybe json
    • Source data from database (MySQL)
    My guess is that reading the text file would be much faster so that I can fetch all the records in one go and then perform search inside array to get relevant results for autocomplete query.

    However, there's another thought that a properly indexed database can outperform approach #1.

    I wish to know if anyone's faced similar situation and what was the optimum solution. How does the efficiency get affected when the number of records is say 50K or 100K?
    Replies
Howdy guest!
Dear guest, you must be logged-in to participate on CrazyEngineers. We would love to have you as a member of our community. Consider creating an account or login.
Replies

  • Kaustubh Katdare

    Administrator • Jun 23, 2016

    FWIW, the server will run SSD; which should make file reading a lot faster. While, I also have the option of having memcache.
    Are you sure? This action cannot be undone.
    Cancel

  • Kaustubh Katdare

    Administrator • Jun 23, 2016

    Tagging #-Link-Snipped-# and #-Link-Snipped-#
    Are you sure? This action cannot be undone.
    Cancel

  • Prasad Ajinkya

    Member • Jun 23, 2016

    You need to factor in the speed of the clients as well. If its a relatively high end client, I'd go with json and do the heavy lifting at the client side 😛

    If its a low end client, then a mysql lookup it is.
    Are you sure? This action cannot be undone.
    Cancel

  • Kaustubh Katdare

    Administrator • Jun 23, 2016

    Prasad Ajinkya
    You need to factor in the speed of the clients as well. If its a relatively high end client, I'd go with json and do the heavy lifting at the client side
    Everything's server side. What would be the use case to transport all the data from server to the client for processing? My thought is that I'd simply send an AJAX request to to server and let it do all the processing.

    With that setup - would you trust json to be speedier than mysql?
    Are you sure? This action cannot be undone.
    Cancel

  • simplycoder

    Member • Jun 25, 2016

    Nice question 😀

    I have not tried this practically, so I won't be clamming this but just my opinion,
    I think json would out perform mysql.

    Cache would play real important role here and I think it would outperform file read as well.

    Even if singleton pattern is correctly implemented for data access, it is likely, that connection handshake time would be higher than that of fetching the data, and if the plan is to hit the database every time a key is entered, it is very likely to be slower (due to db-server <-> app-server handshakes) than a file read.

    There is a third option which can be used, cache the records if they are to be retrieved as master.
    You can put a redis/memcache (or equivalent) and cache the data up, this should outperform both file reads as well as mysql in my opinion.

    One more option, I can think of is, cache the data (I am assuming master data) on server and then cache the data on browser (local storage), so this would help to reduce the trips if the user searches for the same data (This is optimizing thought, but felt to share).
    Are you sure? This action cannot be undone.
    Cancel

  • Kaustubh Katdare

    Administrator • Jun 25, 2016

    simplycoder
    Even if singleton pattern is correctly implemented for data access, it is likely, that connection handshake time would be higher than that of fetching the data,
    That definitely is not the case. We inject delays and trigger the request only after 3 characters. However, if we ever use memcached or redis - we might bring it down to 1 keystroke.

    I'm not very sure if caching data on the client side is a good option - haven't really done this myself. I wonder if the method involves transporting entire data to client; and if that's the case we aren't likely to do that. Connection speeds in India aren't great. Plus, the data is likely to be used just once.

    Maybe server side caching is the key here. Gotta explore.
    Are you sure? This action cannot be undone.
    Cancel
Home Channels Search Login Register