p2asmaven should give more logging/feedback during long operations

[kriegfrj]

I've been playing around with p2asmaven recently in trying to port an OSGI project to Gradle.

There are times where p2asmaven takes several minutes between outputs to the screen. This is a bit disconcerting as it isn't clear if the build has hung or if it is simply taking a long time. In once case I left it running for an hour or two and it wasn't clear to me if it was doing something or if it had hung.

It would be nice if there was at least an option to produce more logging output while it is doing these long operations so that the user knows that something is still happening.

[kriegfrj]

As an example, I just ran p2asmaven on one of the Orbit P2 repositories (http://download.eclipse.org/tools/orbit/downloads/drops/R20170919201930/repository/), and it took 30minutes to complete. For most of that time there was no output to the console, I wasn't sure if it had stopped.

[nedtwigg]

I agree 100%! Unfortunately we just shell out to the eclipse tooling. If there are any progress flags for the eclipse tooling we'd enable them, but I wasn't able to find any. You might be able to speed it up a bit with some of these flags: #41

[kriegfrj]

Hmm, it appears to be the runnable repo task that is stuck - the p2 task finished in 30 minutes and the runnable repo task has been running ever since (an hour at the time of posting). On disk, the size of the original repo is ~300MB, while the runnable repo directory is only 97 MB and has been since very soon after the runnable repo task started. Is this stuck? This is probably a separate issue to the original post, though if the requested feature were available it would probably reveal the answer to this issue...

[nedtwigg]

I just uploaded a potential to fix to 3.11.0-SNAPSHOT, which you can grab like this. Lemme know if it improves your performance or fixes the runnable repo issue.

[hacki11]

i have been there - we use about 10 different p2 repositories with a sum of about 1gb dependencies. During the task of porting from maven to gradle i had to incrementally add dependencies. To avoid redownloading everything again and again i added a nexus p2 proxy and added the 'append' feature onto this plugin.

although gradle provides good incremental build features, in the end we had to do clean builds on our CI. Existing tests and deliverables of our enterprise application did not fit into very well.

Clean builds enforce you to redownload the entire dependency world again and again and we had very time consuming build times. Additionally, restricted proxy settings at CI will add more complexity to this situation.

Now we have a wrapped the p2asmaven configs into a gradle task which downloads the configured dependencies and zips them (p2, p2runnable, maven) into a zip file. This zip file will be extracted in gradle configure phase and a download will not be necessary anymore.

Of course we had to add this big binary file to our vcs but the advantage overweights for us. Dependencies from p2 repositories won't change or need to be added that often, most are available from maven central.

we switched our mixed pde/classic java application from a ant/maven combination to a gradle solution and its worth the hassle!

[kriegfrj]

@nedtwigg, thank you for giving this a try.

I tried this on Friday, but didn't notice any difference. Downloading orbit took about 31 minutes as usual, and "creating runnable repo" is still (allegedly) running after the weekend. Same symptoms - the zipped repo has come in at about 340MB on disk, but the runnable repo has stalled at 97MB.

There seems to be two separate issues here - the root problem, and the lack of logging which makes it hard to recognise (in a timely manner) when the root problem has/is occurring. Would you prefer I open a new issue to track the root problem?

[nedtwigg]

If you're able to create a snippet that reproduces the "infinite runnable repo" problem, that's definitely worth its own issue.

[liblit]

Note that Travis CI builds time out if ten minutes go by with no output. There are Travis-specific ways to increase that limit, but unless a developer learns about those features, they may find that many of their CI builds fail during slow, silent p2asmaven downloads. Progress feedback out-of-the-box would be much appreciated in this scenario.