Configurable (or just much longer?) health-check timeout in local mode

[athewsey]

Describe the feature you'd like

Today, local-mode endpoint deployment uses a hard-coded health check time-out of 120s for the container to become healthy.

This does not appear to be consistent with the start-up requirements for actual SageMaker endpoints, and even if it was, it may not be appropriate to assume local environments have similar network bandwidth or compute capabilities to target instance types.

How would this feature be used? Please describe.

I'm currently testing a use case with large (e.g. ~5GB+) model archives, and finding local mode deployment fails due to this healthcheck time-out, even though actual SageMaker endpoint deployments succeed without any issue.

If the default timeout was significantly longer, I think it should work okay. If the default timeout was configurable somehow, I could force it to wait longer for my use case.

Describe alternatives you've considered

Possible options could include:

• Extending the timeout
• Making the timeout configurable
• Somehow excluding tarball download/extract time from the coverage of the timeout check
• Supporting decompressed local folders as model_data targets for local models/endpoints - instead of requiring S3/tarball.

Additional context

N/A

[cceyda]

(I saw this just as I was about too open the same issue, So here:)
Describe the bug
While using local mode if the container takes a bit long to spin up because there are a lot of requirements being pip installed (or it could just be a slow pip download speed day)
It timeouts due to the HEALTH_CHECK_TIMEOUT_LIMIT here

sagemaker-python-sdk/src/sagemaker/local/entities.py

Line 44 in 8fbc1c6

HEALTH_CHECK_TIMEOUT_LIMIT = 120

while the container keeps on being spun up and start/remain in the background eating up resources. I have to kill it manually everytime 😞

To reproduce
Really just a basic example script with lots of packages in requirements.txt

from sagemaker.local import LocalSession
import sagemaker
sess = LocalSession()
sess.config = {'local': {'local_code': True}}
role = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'

from sagemaker.pytorch import PyTorchModel

model = PyTorchModel(
    source_dir=source_dir,
    entry_point="inference.py",
    model_data=model_data,
    role=role,
    framework_version="1.12",
    py_version="py38"
)
instance_type="local"
predictor = model.deploy( endpoint_name=endpoint_name,
    initial_instance_count=1, instance_type=instance_type, 
)

Expected behavior
Allow timeout to be set by a parameter & if timeout clean up the created container.

Screenshots or logs

System information
A description of your system. Please provide:

• SageMaker Python SDK version: 2.125.0
• Framework name (eg. PyTorch) or algorithm (eg. KMeans): PyTorch (but probably true for all containers)
• Framework version: 1.12 (all)
• Python version: 3.8
• CPU or GPU: both
• Custom Docker image (Y/N): N

Additional context
Currently solving it by monkey patching the timeout. (But if it does timeout the running container in the background still remains)

from sagemaker.local import entities
entities.HEALTH_CHECK_TIMEOUT_LIMIT=9000000