Improve module version purge on managed dependencies auto-upgrade

[AnatoliB]

When requirements.psd1 specifies a range of module versions (for example, { Az = 2.* }), the PowerShell worker compares the currently installed module version with the highest version available on the PowerShell Gallery. If the Gallery contains a higher version, the worker removes the installed version and installs the one from the Gallery. For example, if v2.3 is installed and v2.4 is available, the worker removes v2.3 immediately.

The problem with this approach is that other worker instances may be currently running and using the older version. As a result, either removing the module files will fail at this point, or the worker will encounter module files missing unexpectedly, and hit errors that will be hard to diagnose.

On the other hand, never removing old module versions is also not a good option, as the old versions will continue piling up for modules that are updated on the PowerShell Gallery on the regular basis.

So, in order to ensure smooth transition from older module versions, come up with more conservative criteria, for example:

• Keep at least N latest module versions; and
• Remove module versions that were installed more than M days ago.
  Make the parameters configurable (per function app?), decide on good default values.

[TylerLeonhardt]

Keep at least N latest module versions

The modules end up in the customer's storage account, right? We should be mindful to not keep too many versions around as it impacts the user's storage account space

[TylerLeonhardt]

I could also see a potential race condition here...

If a worker is currently updating modules (let's take Az as an example which has many dependencies) And another worker starts an invocation... That that invocation will use a new version of 1 module and an old version of another. If those two modules are not compatible, that could result in potential breakages.

Example:

1. Worker A - updating Az module...
2. Az.Accounts module gets updated
3. Worker B - invokes with new Az.Accounts module and old Az.Compute module
4. Worker B - Old Compute doesn't play nice with new Accounts causing the invoke to fail
5. Worker A - Az.Compute module gets updated

Your proposal is much better than what we have today but I wanted to call this out as a concern. I don't know how this could be solved since workers don't know about other workers.

[AnatoliB]

@TylerLeonhardt Thank you, this is a valid concern. I do have some ideas on how to resolve it, but they will either impact performance or add complexity.

For example, I'm thinking about upgrading the entire module set atomically: download modules to a separate folder outside of PSModulePath, and then make the next worker instance point to this new folder instead of the previous one, so that each worker instance always observes a consistent and immutable set of modules. We can definitely make it work - I've done something similar for a different project. However, this will complicate the folder structure, making it look quite different from what a regular PowerShell user normally observes.

Before going into these complications, I would try to answer a different question: do we even want to solve this problem in Azure Functions? Any PowerShell user may run into this problem by running scripts on a local box and installing/upgrading modules at the same time. So, perhaps this is actually a module design problem, and module authors are supposed to either keep all module versions within the same major version mutually compatible, or give clear guidance to users on how these modules should be upgraded. Azure Functions users who run into these troubles can always opt out of auto-upgrade by specifying exact module versions. Those who rely on auto-upgrade are in fact accepting a certain risk, so they should use this feature only if they trust the module owners enough.

[AnatoliB]

I could also see a potential race condition here...

We've been thinking and discussing different scenarios around this area. We will definitely want to address them better soon, most likely before GA. I will file a separate issue for this.

[AnatoliB]

Addressed by #267