-
Notifications
You must be signed in to change notification settings - Fork 3
Open
Labels
Description
Portable implementation:
- libuv-style, with thread pool for handling blocking file I/O requests
Efficient Implementation:
- Unlike libuv, we want to do all file I/O REALLY asynchronously; to do this, the following ("opportunistic") model is supposed (for those platforms which DO support async file IO, BUT do NOT allow intermixing them with poll()):
- all file read/write calls are initiated asynchronous
- at any point, we know if there is outstanding async file I/O for current thread
- before any call to poll(), if there is outstanding async file I/O, our poll()-ing thread checks (with 0 timeout) if there is any finished async I/O , and if yes - processes it, and repeats this item
- if there is NO outstanding async file I/O - we're waiting in poll() according to current timers (as if there is no file I/O at all)
- if there IS outstanding async file I/O - we're merely checking poll() (with 0 timeout), and are processing whatever we can process; then go to "before any call to poll()" item
- if poll() returned that there is no socket I/O, BUT there is still outstanding async file I/O - we're waiting for file I/O, and are processing it, then going back to "before any call to poll()" item
- this is based on an assumption that file I/O is fast
- we also MAY want to limit wait on async file I/O to approx 10-20ms just in case (it should happen REALLY rarely) - and to go on another round of poll() if timeout is exceeded
- alternatively, we may also want to split "big" writes (larger than 1MB) into smaller chunks (so that each chunk is fast) - with 2-3 chunks outstanding at each moment to run big writes smoothly
- on the client-side, we MIGHT also want to use separate threads for read/writes from/to "slow" (USB) drives.
- this is based on an assumption that file I/O is fast
- all file read/write calls are initiated asynchronous
Suggested implementations per platform:
- Linux: "opportunistic" as described above, with poll() or epoll() + AIO
- Windows: "opportunistic" as described above, with WSAPoll() + WaitForMultipleObjects for files (if number of outstanding requests exceeds 64 - it should be ok to wait for first 64).
- FreeBSD: it seems that kqueue() will allow direct non-blocking implementation