Great article :D The rabbithole actually goes even deeper because buffering on the read side is almost always good, but reads are often way more latency-sensitive than writes. Which leads to the paradox that increasing the buffer size / prefetching more with mmapping can lead to worse performance due to the added latency. And decreasing the buffer size can have the same effect due to issuing more calls, so in order to squeeze in the optimal throughput/latency ratio on reads you have to tune it per workload.... or just try a few different buffer sizes and see which one works better.
Writes are the opposite, they're usually not RAW (read-after-write), they're "fire and forget", so you don't really care about latency, but you do care a whole a lot about your writes actually arriving... again two competing parameters because the more you batch the writes, the faster they are but the bigger chance of a torn write or data loss. And unlike reads, writes are not idempotent on the application level so ideally you'd want some logic to atomically write a meaningful amount of data so it won't get torn, even if it crashes.
Now in practice just about no one actually does any of this but it's a fun thought :D
I have committed a sin of starting with the comments without reading an article, assumed it was about reading and writing text by humans and was increasingly confused by your first paragraph.
I tried to translate it as metaphor at first, but it didn't end up being coherent.
Most of that is just due to the fundamentally poor I/O model of synchronous operations used on Linux and other Unix derived systems followed by hardware manufacturers cheating to make that fundamentally poor model look okay.
Essentially all modern high speed hardware is batched and asynchronous. The way to get high speed performance is to batch and asynchronously submit operations then wait for batchs of asynchronous completion. In addition to being more efficient to process, batching also enables "smart" hardware to more efficiently schedule the operations if needed since it has more context.
Batching many of the reads you want upfront reduces the need for as much speculative read buffering because you are more capable of telling it "all/most" of the information you actually want instead of it having to guess that you are going to do linear reads. The reads being asynchronous, which is what the hardware does, allows you to issue speculative reads explicitly. If you intend to read a 1 GB file then the userspace I/O library can fire off 1,000 1MB reads and then you can wait until the "next" desired portion of the data comes back before continuing your processing. This is in contrast to issuing a 1 GB synchronous read followed by your processing which delays your processing by the time of the entire read even though the hardware I/O can happen in parallel.
Essentially, the only remaining case where you then want meaningful speculative read buffering (beyond the default efficient block size of the hardware) is that you either do not understand your access patterns enough to batch them, or you have a data-dependent load pattern which happens to regularly resolve to something within the speculative read buffer. For instance, you read 8-bytes which tells you the offset into the file where the desired record is and that record happens to be nearby almost all of the time.
For writes, what you want is a data-or-time flush strategy which is actually just the flush strategy you should use in basically all cases where you do buffering. Data-based flushing exists to achieve efficient batching. Issuing large amounts of small operations kills you with overhead, so you buffer the operations to attempt to coalesce them into larger operations where the overhead is amortized away. However, if inadequate data is generated to reach the data-based flush condition, then you get large or potentially unbounded latency. You resolve this by also targeting a worst-case acceptable latency in low throughput conditions. In that case your overhead has been "amortized" away as a percentage of your processing time since the processing time was long enough to reach the time-based flush condition. What you are effectively doing is allocating N% of your time to completion to overhead which either fires if you generate enough data to amortize it away or if you generate enough time to amortize it away.
Writes being asynchronous also allows you to resolve the data loss issue (if the hardware cooperates). You fire it off and wait for the hardware to signal persistence of that write. Until you see a asynchronous completion for that asynchronous write, you can not assume it is persisted. You achieve ordering by issuing a write barrier which tells the hardware to signal completion of the write barrier when all writes earlier in the asynchronous queue have signaled completion. This can also be further extended to other layers if desired to allow the detection of various levels of "persistence".
Unfortunately, Linux and other Unix derived systems use the awesomely terrible fsync() model which is antithetical to efficient I/O. Instead of you issuing a write barrier and then getting notified when it completes, allowing you to continue to asynchronously issue subsequent writes, you synchronously wait for all earlier I/O to complete. This serializes execution as you wait ensuring that any attempt at ordering or persisting data is monstrously expensive. In fact, this is so horrible that hardware manufacturers realized they could massively improve their throughput benchmarks by lying about when data is persisted so the fsync() would complete faster ensuring that their throughput benchmarks would not get shot due to data persistence latency which is a different property. With a asynchronous model you can just rapid-fire the operations and the write barriers (which is the way the hardware is designed to operate) allowing you to evaluate both read/write throughput and data persistence latency without confounding them. This is both dramatically more useful and more efficient.
Compare the ease with which you recognize a bicycle and the difficulty of drawing one.
It’s not the mechanics of the pencil alone although that plays a role a well. But even if you could somehow imprint perfect lines like you imagine them, the bicycle and the lightening etc will not be realistic unless you are a trained artist.
I probably don't have the best reference at hand, but reducing language to a handful of brain areas like Wernicke and Broca seems to be in flux recently (while more research seems needed)[0].
This is true even at the CPU cache level so it's probably going to trickle down all the way.
For example Jon G gave a talk at Jane Street about how he had a mysterious performance bug that was later attributed to cores fighting over the same cache line
False sharing is pretty well known isn't it? I learned about it at university. It's the whole reason why the Linux "dwarves" tools were written, so you can check struct field alignment / grouping against cache line size and make sure you don't accidentally have two adjacent fields used by different cores.
In any discussion of read, write & sync, I'd want to talk about write amplification (read/modify/write cycles), and also in Linux the sync_file_range syscall which is a useful way to sync part of a file.
There is plenty of fun to be had with write errors and fsync. One example was so called "fsyncgate" where postgresql stumbled upon fsync issues: https://news.ycombinator.com/item?id=20491965
I think there were some changes in kernel since that but I can't remember what exactly
> the data may have made it to storage, but that doesn’t necessarily mean it’s actually visible, because the metadata necessary to reach it might not have been written yet.
The author should clarify here that this fsync wrinkle only affects newly created files, not updates to existing ones.
I don't know why I'd "(probably) think" they're anything like symmetrical. After all, I have first-hand experience of reading code and writing code; clearly, very different processes.
I am not sure if you mean what you have written, or you mean the opposite, but reading a big file when there are no available free memory pages will flush memory pages from the disk cache, which previously were caching data from files accessed by your user, or by other users.
In high performance applications, you must not let the kernel choose randomly what pages should be invalidated, but you must use madvise for memory_mapped files or posix_fadvise for files accessed with read/write, with appropriate arguments so that cache memory pages that are not needed any more will be freed with enough time in advance before reading more data.
But "cache invalidation" is unconcerned with that. The problem of looking for data and reading the wrong data in the belief that it's right is called "memory corruption". "Cache invalidation" is when you used to have data cached, but you no longer do. And the problem that this causes is that your data takes longer to find, not that it becomes inaccurate.
You'll notice that removing entries from the cache causes exactly the same effects as changing the data that backs them.
I was hoping for something about human reading and writing, but it was about POSIX I/O APIs. Not bad, and I'd go further and point out that `mmap()` is good for reading but not for writing, and why.
One can read very quickly, but writing takes a long time.
And speech is rather annoying because I read faster than people speak. But the worst thing is when people start writing on the blackboard during a lecture because then you have to watch them write slowly.
Great article :D The rabbithole actually goes even deeper because buffering on the read side is almost always good, but reads are often way more latency-sensitive than writes. Which leads to the paradox that increasing the buffer size / prefetching more with mmapping can lead to worse performance due to the added latency. And decreasing the buffer size can have the same effect due to issuing more calls, so in order to squeeze in the optimal throughput/latency ratio on reads you have to tune it per workload.... or just try a few different buffer sizes and see which one works better.
Writes are the opposite, they're usually not RAW (read-after-write), they're "fire and forget", so you don't really care about latency, but you do care a whole a lot about your writes actually arriving... again two competing parameters because the more you batch the writes, the faster they are but the bigger chance of a torn write or data loss. And unlike reads, writes are not idempotent on the application level so ideally you'd want some logic to atomically write a meaningful amount of data so it won't get torn, even if it crashes.
Now in practice just about no one actually does any of this but it's a fun thought :D
I have committed a sin of starting with the comments without reading an article, assumed it was about reading and writing text by humans and was increasingly confused by your first paragraph.
I tried to translate it as metaphor at first, but it didn't end up being coherent.
As did I. My happy impression that I was to be commended for reading slowly was turned upside down by the same realization.
Makes sense. If you are reading something you need it to make progress, if you are writing you are finished.
Most of that is just due to the fundamentally poor I/O model of synchronous operations used on Linux and other Unix derived systems followed by hardware manufacturers cheating to make that fundamentally poor model look okay.
Essentially all modern high speed hardware is batched and asynchronous. The way to get high speed performance is to batch and asynchronously submit operations then wait for batchs of asynchronous completion. In addition to being more efficient to process, batching also enables "smart" hardware to more efficiently schedule the operations if needed since it has more context.
Batching many of the reads you want upfront reduces the need for as much speculative read buffering because you are more capable of telling it "all/most" of the information you actually want instead of it having to guess that you are going to do linear reads. The reads being asynchronous, which is what the hardware does, allows you to issue speculative reads explicitly. If you intend to read a 1 GB file then the userspace I/O library can fire off 1,000 1MB reads and then you can wait until the "next" desired portion of the data comes back before continuing your processing. This is in contrast to issuing a 1 GB synchronous read followed by your processing which delays your processing by the time of the entire read even though the hardware I/O can happen in parallel.
Essentially, the only remaining case where you then want meaningful speculative read buffering (beyond the default efficient block size of the hardware) is that you either do not understand your access patterns enough to batch them, or you have a data-dependent load pattern which happens to regularly resolve to something within the speculative read buffer. For instance, you read 8-bytes which tells you the offset into the file where the desired record is and that record happens to be nearby almost all of the time.
For writes, what you want is a data-or-time flush strategy which is actually just the flush strategy you should use in basically all cases where you do buffering. Data-based flushing exists to achieve efficient batching. Issuing large amounts of small operations kills you with overhead, so you buffer the operations to attempt to coalesce them into larger operations where the overhead is amortized away. However, if inadequate data is generated to reach the data-based flush condition, then you get large or potentially unbounded latency. You resolve this by also targeting a worst-case acceptable latency in low throughput conditions. In that case your overhead has been "amortized" away as a percentage of your processing time since the processing time was long enough to reach the time-based flush condition. What you are effectively doing is allocating N% of your time to completion to overhead which either fires if you generate enough data to amortize it away or if you generate enough time to amortize it away.
Writes being asynchronous also allows you to resolve the data loss issue (if the hardware cooperates). You fire it off and wait for the hardware to signal persistence of that write. Until you see a asynchronous completion for that asynchronous write, you can not assume it is persisted. You achieve ordering by issuing a write barrier which tells the hardware to signal completion of the write barrier when all writes earlier in the asynchronous queue have signaled completion. This can also be further extended to other layers if desired to allow the detection of various levels of "persistence".
Unfortunately, Linux and other Unix derived systems use the awesomely terrible fsync() model which is antithetical to efficient I/O. Instead of you issuing a write barrier and then getting notified when it completes, allowing you to continue to asynchronously issue subsequent writes, you synchronously wait for all earlier I/O to complete. This serializes execution as you wait ensuring that any attempt at ordering or persisting data is monstrously expensive. In fact, this is so horrible that hardware manufacturers realized they could massively improve their throughput benchmarks by lying about when data is persisted so the fsync() would complete faster ensuring that their throughput benchmarks would not get shot due to data persistence latency which is a different property. With a asynchronous model you can just rapid-fire the operations and the write barriers (which is the way the hardware is designed to operate) allowing you to evaluate both read/write throughput and data persistence latency without confounding them. This is both dramatically more useful and more efficient.
oh i thought this would be about language and how the brain works, that article sounds more interesting to be honest sorry
Indeed, that's what I also thought this would be about; What a bummer.
Someone should post that article instead, lol. I'd love to discuss why I am able to comprehend and read a language almost fully but cannot speak it.
Production is different from reception.
Compare the ease with which you recognize a bicycle and the difficulty of drawing one.
It’s not the mechanics of the pencil alone although that plays a role a well. But even if you could somehow imprint perfect lines like you imagine them, the bicycle and the lightening etc will not be realistic unless you are a trained artist.
Search for "Broca and Wernicke" to learn more about the distinct areas of the brain responsible for producing vs processing language.
I probably don't have the best reference at hand, but reducing language to a handful of brain areas like Wernicke and Broca seems to be in flux recently (while more research seems needed)[0].
[0] https://psycnet.apa.org/fulltext/2024-11707-001.html
Sure - just a decent place to start
Or even something about how writing usefully on a topic requires higher expertise than reading and understanding what was written.
or how if you want to write better, read more
I have a treat for you. A book on the history and development of the English language:
Highly Irregular: Why Tough, Through, and Dough Don’t Rhyme—And Other Oddities of the English Language
[dead]
This is true even at the CPU cache level so it's probably going to trickle down all the way.
For example Jon G gave a talk at Jane Street about how he had a mysterious performance bug that was later attributed to cores fighting over the same cache line
https://youtu.be/tND-wBBZ8RY?is=624w3YoDN2QYBWuD
False sharing is pretty well known isn't it? I learned about it at university. It's the whole reason why the Linux "dwarves" tools were written, so you can check struct field alignment / grouping against cache line size and make sure you don't accidentally have two adjacent fields used by different cores.
In any discussion of read, write & sync, I'd want to talk about write amplification (read/modify/write cycles), and also in Linux the sync_file_range syscall which is a useful way to sync part of a file.
There is plenty of fun to be had with write errors and fsync. One example was so called "fsyncgate" where postgresql stumbled upon fsync issues: https://news.ycombinator.com/item?id=20491965
I think there were some changes in kernel since that but I can't remember what exactly
> the data may have made it to storage, but that doesn’t necessarily mean it’s actually visible, because the metadata necessary to reach it might not have been written yet.
The author should clarify here that this fsync wrinkle only affects newly created files, not updates to existing ones.
(2015)
I don't know why I'd "(probably) think" they're anything like symmetrical. After all, I have first-hand experience of reading code and writing code; clearly, very different processes.
Yep, reading something doesn't invalidate your caches. Or everyone else's caches.
I am not sure if you mean what you have written, or you mean the opposite, but reading a big file when there are no available free memory pages will flush memory pages from the disk cache, which previously were caching data from files accessed by your user, or by other users.
In high performance applications, you must not let the kernel choose randomly what pages should be invalidated, but you must use madvise for memory_mapped files or posix_fadvise for files accessed with read/write, with appropriate arguments so that cache memory pages that are not needed any more will be freed with enough time in advance before reading more data.
It can evict other things for space reasons, sure. But it doesn't make their contents invalid.
What's the difference between erasing a cache and invalidating it?
One doesn't have a piece of data, and the other has incorrect data.
But "cache invalidation" is unconcerned with that. The problem of looking for data and reading the wrong data in the belief that it's right is called "memory corruption". "Cache invalidation" is when you used to have data cached, but you no longer do. And the problem that this causes is that your data takes longer to find, not that it becomes inaccurate.
You'll notice that removing entries from the cache causes exactly the same effects as changing the data that backs them.
I was hoping for something about human reading and writing, but it was about POSIX I/O APIs. Not bad, and I'd go further and point out that `mmap()` is good for reading but not for writing, and why.
I was thinking 'brains'; you read something, well, ok, you write something, Boom that thing is sticking!
I was thinking human speed
One can read very quickly, but writing takes a long time.
And speech is rather annoying because I read faster than people speak. But the worst thing is when people start writing on the blackboard during a lecture because then you have to watch them write slowly.
(2015)