Build instructions

[NicoHood]

I try to build arduino-builder 1.5.1 for archlinux. It looks like the build system and folder structure changed again. I got the following error when building:

==> Starting build()...
# github.com/jstemmer/go-junit-report
loadinternal: cannot find runtime/cgo
src/github.com/arduino/arduino-builder/main.go:47:2: cannot find package "github.com/arduino/arduino-cli/arduino/cores" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/arduino/cores (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/arduino/cores (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:48:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:49:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder/grpc" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder/grpc (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder/grpc (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:50:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder/i18n" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder/i18n (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder/i18n (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:51:2: cannot find package "github.com/arduino/arduino-cli/legacy/builder/types" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-cli/legacy/builder/types (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-cli/legacy/builder/types (from $GOPATH)
src/github.com/arduino/arduino-builder/main.go:55:2: cannot find package "github.com/sirupsen/logrus" in any of:
	/usr/lib/go/src/github.com/sirupsen/logrus (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/sirupsen/logrus (from $GOPATH)
==> ERROR: A failure occurred in build().

Is the readme build instruction up to date? Do you know anything I have missed to build it properly? I guess it must be something obvious that I missed.

[cmaglie]

The structure it's not changed from 1.5.0, but the README is indeed outdated.

To build you just need the latest version of go and run go build, no need to setup GOPATH or something like that, this is no more needed once we migrated to go modules.

Here a transcript from a clean build:

:~/tmp$ git clone git@github.com:arduino/arduino-builder.git
Clone in 'arduino-builder' in corso...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (27/27), done.
remote: Total 6260 (delta 19), reused 29 (delta 11), pack-reused 6222
Ricezione degli oggetti: 100% (6260/6260), 1.48 MiB | 3.20 MiB/s, fatto.
Risoluzione dei delta: 100% (4071/4071), fatto.
:~/tmp$ cd arduino-builder/
:~/tmp/arduino-builder$ go build
:~/tmp/arduino-builder$ file arduino-builder 
arduino-builder: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, not stripped

[NicoHood]

Okay, thanks for that hint.

Just to understand better:
Why are the additional steps not required anymore, and what is the benefit? I am asking, because I want to package it with the best quality for the distribution and users. I've managed to also compile it by adding another go get github.com/arduino/arduino-cli.

Where is the hardware folder gone? I normally edit the platforms.txt files so they point to the distribution provided arduino-ctags version, but it seems it has moved somewhere. But where? :-)

[cmaglie]

Why are the additional steps not required anymore, and what is the benefit?

The benefit is that the dependent packages are listed here:
https://github.com/arduino/arduino-builder/blob/master/go.mod

and each package is listed with a specific git revision, so the build is reproducible. When you run go build with go modules enabled you're sure that the project is built with the correct dependencies.

If you run go get github.com/arduino/arduino-cli you're, instead, downloading and building with the current master of the arduino-cli package (probably, in this exact moment, it corresponds to the correct version, but if you re-run it again in a month, or even tomorrow, it may be not, depending on when the next commit comes in in the arduino-cli project).

The hardware folder is gone because it just contained the old general platform.txt that is now integrated in the new builder, we don't need it anymore.

[NicoHood]

@cmaglie Thank you for your explanation.

I've seen in the PR, that arduino-builder is now deprecated. It looks like a lot has changed recently, and I am not up to date on how to properly package arduino. I kindly ask you, if you can give me a few hints what I need to know about the recent changes.

From my understanding arduino-builder is no longer needed for the current ide, and I should rather package arduino-cli? How does that integrate with the new IDE? You also mentioned that platform.txt no longer exists, but how would I reference the avr-gcc binaries or the arduino-ctags binary?

It would help me a lot to provide a quality good arduino package for archlinux, which is quite outdated for a while (I am so sorry, that was my fault!).

[matthijskooijman]

From my understanding arduino-builder is no longer needed for the current ide,

AFAIU, the current IDE still uses arduino-builder, but the main code for that has all moved to the arduino-cli repo and development happens there. Possibly Arduino create has switched to calling arduino-cli internally already (not sure) and AFAIU this should happen for the java IDE in the future as well. Until then, this repo still serves to build the arduino-builder binaries.

You also mentioned that platform.txt no longer exists, but how would I reference the avr-gcc binaries or the arduino-ctags binary?

I believe this only referred to the platform.txt that was previously shipped along arduino-builder and contained some default values (mostly for compatibility with cores that were created before arduino-builder exists, I suspect). You will still need a platform.txt in the hardware / core directories for your cores.

As for packaging: I think that the arduino/Arduino repo is still the starting point. That has an antfile that will download/collect all needed components (though you probably want to refer to existing packages rather than bundle them, but the antfile should at least show you what's needed for a normal IDE build).

[cmaglie]

@NicoHood don't worry, just package as always, the Arduino IDE still uses arduino-builder (we are phasing it out, but it's not done yet, it will take some time).

There are no major changes in the way the build is done, even if the arduino-builder now is a wrapper of arduino-cli from the outside almost nothing has changed.

[NicoHood]

The following test failed:

go: github.com/nats-io/nats-server/server tested by
	github.com/nats-io/nats-server/server.test imports
	github.com/nats-io/go-nats: github.com/nats-io/go-nats@v1.8.1: parsing go.mod:
	module declares its path as: github.com/nats-io/nats.go
	        but was required as: github.com/nats-io/go-nats

Can that be fixed somehow?

[NicoHood]

You will still need a platform.txt in the hardware / core directories for your cores.

Looks like you are correct:

Picked up JAVA_TOOL_OPTIONS: 
Set log4j store directory /home/user/.arduino15
java.io.FileNotFoundException: /usr/share/arduino/hardware/platform.keys.rewrite.txt (No such file or directory)
WARNING: Error loading hardware folder /usr/share/arduino/hardware/archlinux-arduino
  No valid hardware definitions found in folder archlinux-arduino.
java.io.FileNotFoundException: /usr/share/arduino/hardware/platform.keys.rewrite.txt (No such file or directory)
java.io.FileNotFoundException: /usr/share/arduino/hardware/platform.keys.rewrite.txt (No such file or directory)
java.lang.IllegalStateException: Menu has no enabled items
	at processing.app.Base.selectFirstEnabledMenuItem(Base.java:1665)
	at processing.app.Base.rebuildBoardsMenu(Base.java:1504)
	at processing.app.Base.<init>(Base.java:277)
	at processing.app.Base.main(Base.java:150)

But where can I find those platform.txt files? I mean I could use those "old" files from previous releases, but I expect those files to be hosted somewhere. If they are completely removed I suggest adding them back to this arduino-builder repository.

And what about the java version? Do I need to force java8 or will arduino update/support newer java versions? arduino/Arduino#8903

[cmaglie]

github.com/nats-io/nats-server/server

uh... how did you run tests on this library?

It seems that you are running tests on a (deep) dependency of arduino-builder, but this is not a common practice, I don't think this is what you want.

There are basically no more direct tests on this repository for arduino-builder because we moved them into arduino-cli. Maybe you can try to run tests just on the package github-com/arduino/arduino-cli that is now used by arduino-builder, but it's still a bit weird (becuase then: why just arduino-cli and not all the dependencies...?)

IMHO you can safely skip the test here.

[cmaglie]

But where can I find those platform.txt files? I mean I could use those "old" files from previous releases, but I expect those files to be hosted somewhere. If they are completely removed I suggest adding them back to this arduino-builder repository.

Let me check this one.

[cmaglie]

It seems you're right @NicoHood, I'll check this for the next release of the arudino-builder. In the meantime I've readded the file in the git repo, is this sufficient for you to complete the build?

[NicoHood]

I ran the tests using go test -timeout 60m ..., maybe that is the problem. I will try to fix that.

Adding the hardware folder is a very nice idea. I think you should also add the platform.txt, not only the "rewrite" file. You can tag a new point release or I can add them manually to the package. Depends on if you want to tag a new release or not.