Skip to content

Add support for fs.ReadStream/fs.WriteStream #60

@nobugs-hare

Description

@nobugs-hare

Portable implementation:

  • libuv-style, with thread pool for handling blocking file I/O requests

Efficient Implementation:

  • Unlike libuv, we want to do all file I/O REALLY asynchronously; to do this, the following ("opportunistic") model is supposed (for those platforms which DO support async file IO, BUT do NOT allow intermixing them with poll()):
    • all file read/write calls are initiated asynchronous
      • at any point, we know if there is outstanding async file I/O for current thread
    • before any call to poll(), if there is outstanding async file I/O, our poll()-ing thread checks (with 0 timeout) if there is any finished async I/O , and if yes - processes it, and repeats this item
    • if there is NO outstanding async file I/O - we're waiting in poll() according to current timers (as if there is no file I/O at all)
    • if there IS outstanding async file I/O - we're merely checking poll() (with 0 timeout), and are processing whatever we can process; then go to "before any call to poll()" item
    • if poll() returned that there is no socket I/O, BUT there is still outstanding async file I/O - we're waiting for file I/O, and are processing it, then going back to "before any call to poll()" item
      • this is based on an assumption that file I/O is fast
        • we also MAY want to limit wait on async file I/O to approx 10-20ms just in case (it should happen REALLY rarely) - and to go on another round of poll() if timeout is exceeded
        • alternatively, we may also want to split "big" writes (larger than 1MB) into smaller chunks (so that each chunk is fast) - with 2-3 chunks outstanding at each moment to run big writes smoothly
        • on the client-side, we MIGHT also want to use separate threads for read/writes from/to "slow" (USB) drives.

Suggested implementations per platform:

  • Linux: "opportunistic" as described above, with poll() or epoll() + AIO
  • Windows: "opportunistic" as described above, with WSAPoll() + WaitForMultipleObjects for files (if number of outstanding requests exceeds 64 - it should be ok to wait for first 64).
  • FreeBSD: it seems that kqueue() will allow direct non-blocking implementation

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions