I ran into the IndexedDB slowness problem while adding support for it to Converse.js XMPP messenger[1].
I'm using localForage which lets me persist data to either sessionStorage, localStorage or IndexedDB (depending on user configuration), however writes to IndexedDB were excruciating slow.
I didn't want to give up on it however, because localStorage has a storage limit that power users were constantly coming up against, and IndexedDB is also necessary for progressive web apps.
Eventually I solved the problem by batching IndexedDB writes with a utility function I wrote called mergebounce[2]. It combines Lodash's "debounce" and "merge" functions to combine multiple function calls into a single one, while keeping and merging the data that was passed to the individual function calls.
That way, you can call a function to persist data to IndexedDB many times, but it only executes once (after a small delay). I wrote a blog post about this approach[3]. It's generic enough that anyone else who uses localForage for IndexedDB could use it.
Depends how complex your needs get, IMO. localStorage is a key/value store, IndexedDB is a (basic) database. (localStorage also ties up the main thread whenever its in use, which also becomes relevant if you're using it a ton)
Looks like only transactions are "slow", where "slow" means 2ms, which is perfectly acceptable since it needs to wait for disk commit.
So the problem is that the author is using it wrong, since you only need to wait for transaction commit when you need guaranteed durability, and you only need guaranteed durability when you are communicating success to the user or some other system, which only needs to happen every frame, which is 16ms at 60fps which is plenty of time to wait for a 2ms transaction commit (network communication can also be throttled to the framerate).
So in practice all you need to do is batch writes in a single transaction per frame and there is no problem.
And of course if you don't need durability you can just keep the data in the JavaScript heap (and use inter-tab communication if needed).
I'm more interested in read speed than write speed. I have about 2 MB of data that I fetch, parse and transform into a nested object for easy look-up by various types of keys. It consists of 6 other objects, and I'd guess it's < 50 MB in total size.
In my brief experiment, it was 12% faster to read from the web Cache API [1], re-parse and re-transform that nested object than to read the fully transformed object using IndexedDB via idb-keyval [2]. That surprised me! I went on to learn that IndexedDB does a structured clone as part of such reads, which I suspect is the main cause of slowness in my use case.
Related commits to reproduce that finding are in [3], specifically [4].
The conclusion of this article is "do not use IndexedDB as a database", but the reasoning is "if you bulk-insert by creating a transaction for every operation, it's slow".
If you were using IndexedDB as a database, you wouldn't bulk-insert data in this way, as it's the slowest way to do it.
The conclusion should be "use IndexedDB as you would use a database".
It isn't how you would do it if you were bulk loading a bunch of pre determined data, but if it were a bunch of random business transactions in an OLTP scenario that is pretty much how it would happen. The fact that IndexedDB scales poorly in the number of transactions it can handle is an important data point when considering what tech to use.
Well, that hypothetical OLTP database almost certainly has many concurrent clients, right? Your single-threaded JavaScript app is not going to be generating the same usage pattern.
I am still trying to figure out why anyone wants a database in their document display program. As far as I am concerned browsers were complete a while ago, and devs are just adding random stuff so they can keep getting a paycheck.
Unless I missed it, I assume the article is talking about the Chrome implementation of IndexedDB. As far as I know the performance issue with their implementation vs. say, Firefox’ one is a known issue[1].
Other comments mention using batch operations to speed-up IDB but maybe Chrome could also fix their implementation in the first place. It seems, though, that the Chrome team had no intention on improving the write performance of IDB as of 2019[2].
Edit: it seems that Chrome team did improve the performance of IDB since 2019 but it is still not at Firefox level of performance.
I don’t think you should be expecting a local in-memory db to require batching for reasonable performance; the main reason it’s a problem for normal DB’s is the network — you’re constantly waiting for the roundtrip.
Which you obviously still have for an in-memory DB, but you should expect it to require a lot more picked low-hanging fruit before that becomes your bottleneck.
I am fairly certain that the reason you want to batch writes in a normal database is not because of RTT latency, but rather transaction overhead; e.g. indexes have to be updated only once, rather than thousands of times. Disks have to be flushed only once, etc.
I am fairly certain the issues are similar for IndexedDB.
Each transaction typically has a non trivial amount of overhead as committing it requires you to verify that there wasn’t a conflict.
Why would you expect that you wouldn’t need to do this just because it’s an in-memory database? You might still be modifying it concurrently through multiple tabs open to the same site modifying the same DB or because you are interleaving “concurrent” actions for some reason within a page. Has nothing to do with I/O to my knowledge.
Try this experiment out with any transactional in-memory tables on any database and they should all show some amount of degradation (although maybe not necessarily quite as severe as IndexDB - 10k inserted rows/s seems below what SQLite should be putting out).
You can "open" multiple stores in a transaction so there's no need to have dozens for a dozen stores. There's some onus on you to work out the batching as IndexedDB is quite low level for what we usually get in the browser.
> But FileSystem API seems to finally getting broad support.
I'm not seeing much beyond Chrome & it's associates for it?
That said that could just be because "filesystem" as a word is mentioned in a dozen different web APIs from allowing file input reading locally, to the deprecated API chrome had for local file access, to the new thing that came out of "project Fugu" for PWAs to have native file access and I'm missing which one is which in terms of support & docs.
Thanks, I will look into that. (Is this a recent addition to the API, or did I just missed that option from the beginning? My db code is pretty much unchanged since a few years)
And yes, file system api is confusing, I was also unsure for a moment to have looked at the wrong API, but I was refering to there, the local sandbox filesystem:
Which says it mostly works now, but I never worked with it, because of chrome only, despite it seemed so much better suited for my use case than indexedDB.
> When the window fires the beforeunload event we can assume that the JavaScript process is exited any moment and we have to persist the state. After beforeunload there are several seconds time which are sufficient to store all new changes. This has shown to work quite reliable.
On the contrary, I've found onbeforeunload far from reliable (albeit my use-case was a bit different). You should probably not rely on it for anything important.
This has been my experience as well. At a prior job, we experimented with deferring reporting/tracking functions until the browser was idle, and then firing the remaining tracking events on unload and found that a non-trivial number of events were just dropped (I believe due to this issue).
Funny, because that was almost exactly my use-case as well. I guess sending network requests during the onload event doesn't work well, but maybe for local things it works better, who knows.
One thing I don't see called out as an option: CacheStorage (window.caches). Just store your data as Response. Fast, easy, available in Worker scope...
Doesn't solve rich data though as neatly as IndexedDB does.
Because IndexedDB uses structured clone, it can store javascript data as is, which includes storing correctly a number of interesting/useful types, including ArrayBuffers, Files/Blobs, CryptoKey instances (useful to avoid needing export flag on them), the cost of this is the performance hit compared to JSON.parse on a string.
Also the cache is less persistent than IndexedDB when storage pressure increases.
Not to take away from the points made in the article, but isn't inserting, hundreds or thousands of documents into IndexedDB at once kinda of an edge case?
I think that developers are constantly surprised and frustrated when writing data to IndexedDB takes much longer than downloading it. High transaction cost turns out to be one of several performance surprises with Chrome IndexedDB. If a developer discovers these one-by-one, then they will probably need to redesign a few times if they want IndexedDB to support a data-intensive or offline application.
I get the reasoning behind the browser makers' decision, but I really wish Web SQL had become the ubiquitous option. SQLite doesn't make a standard, but pragmatically I think it would have been a better option than how IndexedDB turned out.
A declarative storage syntax for a declarative display engine. Seems a good fit.
I wonder why they don't mention dexie with all of the other options. Like they say, batched operations solves this handily and Dexie has great support for them and it's really darn fast.
I'm using localForage which lets me persist data to either sessionStorage, localStorage or IndexedDB (depending on user configuration), however writes to IndexedDB were excruciating slow.
I didn't want to give up on it however, because localStorage has a storage limit that power users were constantly coming up against, and IndexedDB is also necessary for progressive web apps.
Eventually I solved the problem by batching IndexedDB writes with a utility function I wrote called mergebounce[2]. It combines Lodash's "debounce" and "merge" functions to combine multiple function calls into a single one, while keeping and merging the data that was passed to the individual function calls.
That way, you can call a function to persist data to IndexedDB many times, but it only executes once (after a small delay). I wrote a blog post about this approach[3]. It's generic enough that anyone else who uses localForage for IndexedDB could use it.
1. https://conversejs.org 2. https://github.com/conversejs/mergebounce 3. https://opkode.com/blog/2021-05-05-mergebounce-indexeddb/