Make use of URLConnection cache configurable in PlexusIoURLResource

[rbjorklin]

See #2 on why the cache was initially disabled.

Maven-assembly-plugin depends on plexus-io and having the cache enabled does wonders for the build performance when building a jar-with-dependencies and the dependencies are fairly large such as eclipse-collections.

EDIT: https://github.com/codehaus-plexus/plexus-io/blob/master/src/main/java/org/codehaus/plexus/components/io/resources/PlexusIoURLResource.java#L42

[michael-o]

Please read my comment at #2.

[plamentotev]

Hi,

How does enabling the cache speeds up your build? Does that means that without the cache a jar is downloaded multiple times(during the build not necessary by the Assembly plugin)? My point is that if a jar is downloaded multiple times, when it is perfectly safe to use cache, then the local .m2 cache should be used instead. So maybe that is the root cause of the issue.

[rbjorklin]

@plamentotev My current understanding is that the file is kept in memory and each class file is read from the cached jar. When the cache isn't used the 9.6MB eclipse-collections jar is read from disk once per class file which takes unnecessary time. The build does not download the file multiple times, the build works and takes the same amount of time with the network cable unplugged.

[michael-o]

@rbjorklin Did you check your assumption with VisualVM or similar?

[rbjorklin]

@michael-o With the help of YourKit we could see that the maven build spent a majority of it's time in this class and we decided to try turning the cache on. We're currently using a snapshot build of plexus-io with the cache enabled and seeing good build performance with maven-assembly 3.1.0.

[michael-o]

@rbjorklin Thanks for the update. Willing to review a PR.

[plamentotev]

@rbjorklin thanks for the information. Plexus Archiver uses PlexusIoZipFileResourceCollection to add the content of the Jar dependencies into the resulting archive. It uses separate PlexusIoURLResource for each entry in the Jar file and that leads to the performance penalty you see. I just didn't expect the difference to be so huge if the Jar file is stored on the file system.

If Plexus Archiver uses PlexusArchiverZipFileResourceCollection the issue with opening the Jar file for each entry would be solved. Would you please try if that solves you performance issue? It easy to make the change. Here is the diff based on Plexus Archiver 3.6.0:

diff --git a/src/main/resources/META-INF/plexus/components.xml b/src/main/resources/META-INF/plexus/components.xml
index 71d47ac..1c613b7 100644
--- a/src/main/resources/META-INF/plexus/components.xml
+++ b/src/main/resources/META-INF/plexus/components.xml
@@ -321,7 +321,7 @@
       <role>org.codehaus.plexus.components.io.resources.PlexusIoResourceCollection</role>
       <role-hint>jar</role-hint>
       <!-- there is no implementation of PlexusIoJarFileResourceCollection, but PlexusIoZipFileResourceCollection will do the job -->
-      <implementation>org.codehaus.plexus.archiver.zip.PlexusIoZipFileResourceCollection</implementation>
+      <implementation>org.codehaus.plexus.archiver.zip.PlexusArchiverZipFileResourceCollection</implementation>
       <instantiation-strategy>per-lookup</instantiation-strategy>
     </component>
     <component>

As far I know PlexusArchiverZipFileResourceCollection is the better implementation but there are some IT failing (but that was some time ago) preventing it from being the default. I guess its worth investigating if PlexusArchiverZipFileResourceCollection could be used for all ZIP based formats - Jar, War, etc

[michael-o]

@plamentotev What is the difference between PlexusIoZipFileResourceCollection and PlexusArchiverZipFileResourceCollection in brief?

[plamentotev]

@michael-o if you ask why there are two implementations - all I know is this comment.

If you ask what is the difference between the implementations:

• PlexusIoZipFileResourceCollection uses org.apache.commons.compress.archivers.zip.ZipFile to get the list of all entries inside the ZIP file, but the InputStream for each entry is retrieved using PlexusIoURLResource (each entry can be represented as jar:{fileURL}!/{entryName}). But that means the ZIP headers are parsed for each entry to locate it inside the ZIP file.
• PlexusArchiverZipFileResourceCollection also uses org.apache.commons.compress.archivers.zip.ZipFile to get the list of all entries inside the ZIP file, but it uses it to retrieve the InputStream for each entry as well. So the extra work to open the ZIP file and read the headers for each entry is avoided.

At first sight looks like they should be interchangeable, but maybe there is some subtle difference in the behavior. PlexusArchiverZipFileResourceCollection is used for ZIP files, so in the base case it works. PlexusIoZipFileResourceCollection is used for Jar, War, etc.

[michael-o]

That's really confusing because they are all ZIP files after all.

[plamentotev]

Yes, that is why I think is better to replace PlexusIoZipFileResourceCollection with PlexusArchiverZipFileResourceCollection for all ZIP formats than fixing the PlexusArchiverZipFileResourceCollection performance. I've created codehaus-plexus/plexus-archiver#90 so we can track it.

That being said, making the cache configurable may still make sense if PlexusIoURLResource have other uses.

[rbjorklin]

@plamentotev Just want to make sure I understood you correctly. The change you proposed was in the maven-assembly-plugin itself, right? If so then changing to PlexusArchiverZipFileResourceCollection made no performance difference unfortunately. So for now we're sticking with our patched plexus-io.

Any suggestions on a sane way forward?

[plamentotev]

Excuse me, I didn't made myself clear which project I have in mind. The change is in Plexus Archiver( https://github.com/codehaus-plexus/plexus-archiver). The components.xml in the Assembly plugin controls only the sar files and have no effect on jars.

…

On Mon, May 28, 2018, 16:33 Robin ***@***.***> wrote: @plamentotev <https://github.com/plamentotev> Just want to make sure I understood you correctly. The change you proposed was in the maven-assembly-plugin itself, right? If so then changing to PlexusArchiverZipFileResourceCollection made no performance difference unfortunately. So for now we're sticking with our patched plexus-io. Any suggestions on a sane way forward? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMLobagqvApwU0Hm0U62zRZWRncJKqe5ks5t2_ylgaJpZM4UKY2u> .