[tests] NO dropbox, fix locking in Xamarin.ProjectTools

[jonathanpeppers]

Context: https://jenkins.mono-project.com/job/xamarin-android-pr-builder/4408/testReport/junit/Xamarin.Android.Build.Tests/BuildTest/BuildBasicApplicationCheckPdb___Debug/

A random test failure has occurred such as:

Xamarin.Android.Build.Tests.BuildTest.BuildBasicApplicationCheckPdb

System.IO.IOException : Sharing violation on path /Users/builder/.local/share/Xamarin.ProjectTools/PdbTestLibrary.pdb
    at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share, System.Int32 bufferSize, System.Boolean anonymous, System.IO.FileOptions options) [0x0019e] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at System.IO.FileStream..ctor (System.String path, System.IO.FileMode mode, System.IO.FileAccess access, System.IO.FileShare share) [0x00000] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at (wrapper remoting-invoke-with-check) System.IO.FileStream..ctor(string,System.IO.FileMode,System.IO.FileAccess,System.IO.FileShare)
    at System.IO.File.OpenRead (System.String path) [0x00000] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at System.IO.File.ReadAllBytes (System.String path) [0x00000] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at Xamarin.ProjectTools.BuildItem+<>c__DisplayClass71_0.<set_WebContent>b__0 () [0x0004f] in <7ee8601bba4749f49f3dee0ccd7e3cca>:0
    at Xamarin.ProjectTools.XamarinProject+<>c.<Save>b__88_4 (Xamarin.ProjectTools.BuildItem s) [0x00050] in <7ee8601bba4749f49f3dee0ccd7e3cca>:0
    at System.Linq.Enumerable+SelectListIterator`2[TSource,TResult].MoveNext () [0x00048] in <7505d9a21cfc4fc7b41cb4768918fdb4>:0
    at System.Collections.Generic.List`1[T].AddEnumerable (System.Collections.Generic.IEnumerable`1[T] enumerable) [0x00059] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at System.Collections.Generic.List`1[T].InsertRange (System.Int32 index, System.Collections.Generic.IEnumerable`1[T] collection) [0x000f4] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at System.Collections.Generic.List`1[T].AddRange (System.Collections.Generic.IEnumerable`1[T] collection) [0x00000] in <9e4df56871b74651838be0f6a9e1bc80>:0
    at Xamarin.ProjectTools.XamarinProject.Save (System.Boolean saveProject) [0x001ec] in <7ee8601bba4749f49f3dee0ccd7e3cca>:0
    at Xamarin.ProjectTools.ProjectBuilder.Save (Xamarin.ProjectTools.XamarinProject project, System.Boolean doNotCleanupOnUpdate, System.Boolean saveProject) [0x00001] in <7ee8601bba4749f49f3dee0ccd7e3cca>:0
    at Xamarin.ProjectTools.ProjectBuilder.Build (Xamarin.ProjectTools.XamarinProject project, System.Boolean doNotCleanupOnUpdate, System.String[] parameters, System.Boolean saveProject, System.Collections.Generic.Dictionary`2[TKey,TValue] environmentVariables) [0x00001] in <7ee8601bba4749f49f3dee0ccd7e3cca>:0
    at Xamarin.Android.Build.Tests.BuildTest.BuildBasicApplicationCheckPdb () [0x00102] in <afa5582a08a34f8ab071ff115b292110>:0
    at (wrapper managed-to-native) System.Reflection.MonoMethod.InternalInvoke(System.Reflection.MonoMethod,object,object[],System.Exception&)
    at System.Reflection.MonoMethod.Invoke (System.Object obj, System.Reflection.BindingFlags invokeAttr, System.Reflection.Binder binder, System.Object[] parameters, System.Globalization.CultureInfo culture) [0x0003b] in <9e4df56871b74651838be0f6a9e1bc80>:0

BuildTest

Looking at the test which this occurred, we have two tests using
different URLs, but the same filename:

https://www.dropbox.com/s/s4br29kvuy8ygz1/PdbTestLibrary.dll?dl=1
https://dl.dropboxusercontent.com/u/18881050/Xamarin/PdbTestLibrary.dll

The second one 404's! Meaning, this test could only possibly succeed
when the first file has already been downloaded and cached!

This test could certainly fail on a new build bot, depending on the
ordering of the test run.

So that brings up a point... How do we know these dropbox URLs will
always work? We don't.

I went through and uploaded all the files to our Azure storage account
that is already used for the Mono bundle and other build artifacts.

These files will be downloaded from:

https://xamjenkinsartifact.azureedge.net/mono-jenkins/xamarin-android-test/{filename}

I added a new WebContentFileNameFromAzure to BuildItem that can be
used here instead of WebContent. Then the filename can just be
specified, assuming the file has been uploaded.

DownloadedCache

In 88d98aa I added some locking to help concurrent downloads. This
greatly helped test runs on Windows.

However, it was not quite precise enough:

The locking was done upon the URL string, and in this instance the
URLs differ, but the cache filename would be the same.

I moved the locking into the DownloadedCache cache class, where it
should have been in the first place. The locking functionality is the
same, except it will now lock on the cached filename instead of the
URL string.

I also removed a Directory.Exists check that isn't needed.

[jonathanpeppers]

I logged into the macOS build machine and deleted /Users/builder/.local/share/Xamarin.ProjectTools/, we'll see if this fares any better.

[jonathanpeppers]

Oops I think I forgot to upload one file to Azure there is a 404. There are some on-device network tests failing too, probably unrelated.

Uploaded missing files and retrying.