Shared memory data transfer between Functions Host and Python worker#816

Merged

Hazhzeng merged 81 commits intodevfrom

gochaudh/shared_mem_data_transfer

Mar 24, 2021

Commits on Feb 19, 2021

Basic structure laid out for reading parameters from shared memory before passing to the function
Gohar Irfan Chaudhry
committed
Writing output from worker to Shared Memory
Gohar Irfan Chaudhry
committed
Put output from worker into Shared Memory
Gohar Irfan Chaudhry
committed
Free shared memory resources after use
Gohar Irfan Chaudhry
committed
Removed control flag from mmap header
Gohar Irfan Chaudhry
committed
Proto change
Gohar Irfan Chaudhry
committed
Working for blob shared memory data transfer; needs clean up, comments and tests
Gohar Irfan Chaudhry
committed
Changing message for closing mmaps
Gohar Irfan Chaudhry
committed
Support for string datatype for shared memory data transfer
Gohar Irfan Chaudhry
committed
Change to oneof
Gohar Irfan Chaudhry
committed
Use oneof in .proto
Gohar Irfan Chaudhry
committed
Refactoring mmap_handler
Gohar Irfan Chaudhry
committed
Refactoring, cleaning up and adding docstrings
Gohar Irfan Chaudhry
committed
Updating CloseSharedMemoryResourcesResponse usage
Gohar Irfan Chaudhry
committed
Fixing accidental changes to tests/*
Gohar Irfan Chaudhry
committed
Addressing comments
Gohar Irfan Chaudhry
committed
Basic structure laid out for reading parameters from shared memory before passing to the function
Gohar Irfan Chaudhry
committed
Writing output from worker to Shared Memory
Gohar Irfan Chaudhry
committed
Put output from worker into Shared Memory
Gohar Irfan Chaudhry
committed
Free shared memory resources after use
Gohar Irfan Chaudhry
committed
Removed control flag from mmap header
Gohar Irfan Chaudhry
committed
Proto change
Gohar Irfan Chaudhry
committed
Working for blob shared memory data transfer; needs clean up, comments and tests
Gohar Irfan Chaudhry
committed
Changing message for closing mmaps
Gohar Irfan Chaudhry
committed
Support for string datatype for shared memory data transfer
Gohar Irfan Chaudhry
committed
Change to oneof
Gohar Irfan Chaudhry
committed
Refactoring mmap_handler
Gohar Irfan Chaudhry
committed
Refactoring, cleaning up and adding docstrings
Gohar Irfan Chaudhry
committed
Updating CloseSharedMemoryResourcesResponse usage
Gohar Irfan Chaudhry
committed
Fixing accidental changes to tests/*
Gohar Irfan Chaudhry
committed
Addressing comments
Gohar Irfan Chaudhry
committed
Cleaning up, addressing comments
Gohar Irfan Chaudhry
committed
Following same class structure as the shared memory changes made for the host
Gohar Irfan Chaudhry
committed
Moving shared memory data transfer related changes into separate directory
Gohar Irfan Chaudhry
committed
Input error checks for shared memory map ctor
Gohar Irfan Chaudhry
committed
Rebase fix
Gohar Irfan Chaudhry
committed
Trying to make lint happy
Gohar Irfan Chaudhry
committed
Making flake8 happy
Gohar Irfan Chaudhry
committed
Merge branch 'dev' into gochaudh/shared_mem_data_transfer
Gohar Irfan Chaudhry
authored
Fixing more lint issues
Gohar Irfan Chaudhry
committed
Removing the use of __annotations__ to be compatible with Python 3.6, adding FileAccessor tests
Gohar Irfan Chaudhry
committed
Added tests for FileAccessor, SharedMemoryMap, FileAccessorFactory
Gohar Irfan Chaudhry
committed
Adding shared memory test setup for Unix
Gohar Irfan Chaudhry
committed
Adding tests for SharedMemoryManager
Gohar Irfan Chaudhry
committed

Commits on Feb 20, 2021

Adding tests to ensure the dispatcher can invoke the function and send/receive inputs over shared memory using a mock host
Gohar Irfan Chaudhry
committed

Commits on Feb 23, 2021

More tests for worker/dispatcher's use of shared memory
Gohar Irfan Chaudhry
committed
Using truncate instead of writing 0x00 byte manually upon creating mmap. More tests for multiple input/output blobs.
Gohar Irfan Chaudhry
committed
Addressing comments
Gohar Irfan Chaudhry
committed
Adding missing tests and doc strings for tests and their classes
Gohar Irfan Chaudhry
committed
assertEqual -> assertTrue in test_dispose_without_delete_file
Gohar Irfan Chaudhry
committed
Tried with blob trigger function - removed TODO
Gohar Irfan Chaudhry
committed

Commits on Mar 3, 2021

Addressing comments
Gohar Irfan Chaudhry
committed
Merge branch 'dev' into gochaudh/shared_mem_data_transfer
Gohar Irfan Chaudhry
authored

Commits on Mar 4, 2021

Minor const fix
Gohar Irfan Chaudhry
committed

Commits on Mar 8, 2021

Addressing comments
Gohar Irfan Chaudhry
committed
Fixed test - removed unused test functions
Gohar Irfan Chaudhry
committed

Commits on Mar 10, 2021

Addressing comments; caching list of valid directories for later use instead of checking every time
Gohar Irfan Chaudhry
committed
Merge branch 'dev' into gochaudh/shared_mem_data_transfer
Gohar Irfan Chaudhry
authored
Whitespace fix
Gohar Irfan Chaudhry
committed
Cleanup
Gohar Irfan Chaudhry
committed

Commits on Mar 13, 2021

Adding AppSetting to override shared memory directory for Unix - test setup for macOS
Gohar Irfan Chaudhry
committed
Logging subprocess result in Exception
Gohar Irfan Chaudhry
committed
Changed Exception -> IOError
Gohar Irfan Chaudhry
committed
Check shared memory directory AppSetting only for Darwin tests
Gohar Irfan Chaudhry
committed
Fix test cleanup for Darwin
Gohar Irfan Chaudhry
committed
Only split AppSetting list of directories if the AppSetting was found
Gohar Irfan Chaudhry
committed
Fix consts.UNIX_TEMP_DIRS
Gohar Irfan Chaudhry
committed
Typo fix
Gohar Irfan Chaudhry
committed

Commits on Mar 15, 2021

Adding throughput tests for blob input/output using shared memory
Gohar Irfan Chaudhry
committed
Changing branch hardcode to instead use current branch
Gohar Irfan Chaudhry
committed
Creating tests URL from current branch name
Gohar Irfan Chaudhry
committed
Whitespace fixes
Gohar Irfan Chaudhry
committed
Using tests from local dir
Gohar Irfan Chaudhry
committed
Removing the need to use a URL for tests; just use from local directory
Gohar Irfan Chaudhry
committed
Rename env variable to follow convention from host of using FUNCTIONS_
Gohar Irfan Chaudhry
committed

Commits on Mar 19, 2021

Addressing comments
Gohar Irfan Chaudhry
committed
Fixes for running tests on macOS
Gohar Irfan Chaudhry
committed
Merge branch 'dev' into gochaudh/shared_mem_data_transfer
Gohar Irfan Chaudhry
authored
Log which allowed directories for shared memory are being used
Gohar Irfan Chaudhry
committed
Change assert -> raise
Gohar Irfan Chaudhry
committed

Commits on Mar 23, 2021

Merge branch 'dev' into gochaudh/shared_mem_data_transfer
Hazhzeng
authored