idf_tools.py installation failed

[maxgerhardt]

PROBLEM DESCRIPTION

Installation of toolchain fails with message

idf_tools.py installation failed

Subsequently compilers are not found

Compiling .pio/build/cheap_yellow/lib0f1/SPI/SPI.cpp.o
sh: xtensa-esp32-elf-g++: command not found

TO REPRODUCE

Exact reproduction unknown, might need a previous version of pioarduino installed which has triggered the tool installation or not. Occurred for a user with platformio.ini

[env:cheap_yellow]
; has Arduino-ESP32 3.2.1
platform = https://github.com/pioarduino/platform-espressif32/releases/download/54.03.21/platform-espressif32.zip
board = esp32dev
framework = arduino

EXPECTED BEHAVIOUR

Tools are installed normally.

SCREENSHOTS

-/-

ADDITIONAL CONTEXT

As observed in https://community.platformio.org/t/bb-spi-lcd-missing-esp-psram-h/

The TL install tool points to

platform-espressif32/platform.json

Lines 97 to 102 in 9bf8b3f

	"tl-install": {
	"type": "tool",
	"optional": false,
	"owner": "pioarduino",
	"version": "https://github.com/pioarduino/esp_install/releases/download/v5.0.0/esp_install-v5.0.0.zip"
	},

However that package's tools.json is an empty file

Subsequently running the code

platform-espressif32/platform.py

Lines 162 to 172 in 9bf8b3f

	def _run_idf_tools_install(self, tools_json_path: str, idf_tools_path: str) -> bool:
	"""Execute idf_tools.py install command with timeout and error handling."""
	cmd = [
	python_exe,
	idf_tools_path,
	"--quiet",
	"--non-interactive",
	"--tools-json",
	tools_json_path,
	"install"
	]

fails with

C:\Users\Max\temp\psramtest>"C:\Users\Max\.platformio\penv\Scripts\python.exe" C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py  --quiet --non-interactive --tools-json C:\Users\Max\.platformio\packages\tl-install\tools\tools.json install
Traceback (most recent call last):
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 2809, in <module>
    main(sys.argv[1:])
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 2801, in main
    action_func(args)
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 1983, in action_install
    targets = add_and_check_targets(idf_env_obj, args.targets)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 1470, in add_and_check_targets
    targets = parse_targets_arg(targets_str)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 1451, in parse_targets_arg
    targets_from_tools_json = get_all_targets_from_tools_json()
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 1495, in get_all_targets_from_tools_json
    tools_info = load_tools_info()
                 ^^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\packages\tl-install\tools\idf_tools.py", line 1322, in load_tools_info
    tools_info = json.load(f)
                 ^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\python3\Lib\json\__init__.py", line 293, in load
    return loads(fp.read(),
           ^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\python3\Lib\json\__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\python3\Lib\json\decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\Max\.platformio\python3\Lib\json\decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

(Please, remember to close the issue when the problem has been addressed)

[Jason2866]

Close VSC delete the complete folder .platformio Start VSC follow instructions.

[maxgerhardt]

The tool cannot recover from a once deleted toolchain folder.

max@max-VirtualBox:~/temp/esppsram$ rm -rf ~/.platformio/packages/toolchain-xtensa-esp-elf
max@max-VirtualBox:~/temp/esppsram$ rm -rf ~/.platformio/.cache/*
max@max-VirtualBox:~/temp/esppsram$ ~/.platformio/penv/bin/pio run 
Processing cheap_yellow (platform: https://github.com/pioarduino/platform-espressif32/releases/download/54.03.21/platform-espressif32.zip; board: esp32dev; framework: arduino)
-----------------------------------------------------------------------------------------------------------------------------
Tool Manager: Installing https://github.com/pioarduino/registry/releases/download/0.0.1/xtensa-esp-elf-14.2.0_20241119.zip
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: toolchain-xtensa-esp-elf@14.2.0+20241119 has been installed!
Tool Manager: Installing file:///home/max/.platformio/tools/toolchain-xtensa-esp-elf
Tool Manager: toolchain-xtensa-esp-elf@14.2.0+20241119 has been installed!
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp32dev.html
PLATFORM: Espressif 32 (54.3.21) > Espressif ESP32 Dev Module
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (cmsis-dap) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-arduinoespressif32 @ 3.2.1 
 - framework-arduinoespressif32-libs @ 5.4.0+sha.858a988d6e 
 - tool-esptoolpy @ 5.0.0 
 - toolchain-xtensa-esp-elf @ 14.2.0+20241119
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 41 compatible libraries
Scanning dependencies...
No dependencies
Building in release mode
Compiling .pio/build/cheap_yellow/src/main.cpp.o
sh: 1: xtensa-esp32-elf-g++: not found
Compiling .pio/build/cheap_yellow/FrameworkArduino/ColorFormat.c.o
*** [.pio/build/cheap_yellow/src/main.cpp.o] Error 127
sh: 1: xtensa-esp32-elf-gcc: not found
*** [.pio/build/cheap_yellow/FrameworkArduino/ColorFormat.c.o] Error 127
================================================ [FAILED] Took 3.32 seconds ================================================

[Jason2866]

Can not reproduce

Last login: Sat Jul  5 20:02:16 on ttys028
claudia@MacBook-Air-von-Claudia ~ % cd Git/arduino-wifiscan 
claudia@MacBook-Air-von-Claudia arduino-wifiscan % rm -rf ~/.platformio/packages/toolchain-xtensa-esp-elf
claudia@MacBook-Air-von-Claudia arduino-wifiscan % rm -rf ~/.platformio/.cache/* 
zsh: sure you want to delete all the files in /Users/claudia/.platformio/.cache [yn]? y
zsh: no matches found: /Users/claudia/.platformio/.cache/*
claudia@MacBook-Air-von-Claudia arduino-wifiscan % 
claudia@MacBook-Air-von-Claudia arduino-wifiscan % ~/.platformio/penv/bin/pio run
Can not remove temporary directory `/Users/claudia/Git/arduino-wifiscan/.pio/build`. Please remove it manually to avoid build issues
Processing esp-wrover-kit (platform: https://github.com/pioarduino/platform-espressif32/releases/download/54.03.21/platform-espressif32.zip; framework: arduino; board: esp-wrover-kit)
--------------------------------------------------------------------------------
Tool Manager: Installing https://github.com/pioarduino/registry/releases/download/0.0.1/xtensa-esp-elf-14.2.0_20241119.zip
Downloading  [####################################]  100%
Unpacking  [####################################]  100%
Tool Manager: toolchain-xtensa-esp-elf@14.2.0+20241119 has been installed!
Tool Manager: Installing file:///Users/claudia/.platformio/tools/toolchain-xtensa-esp-elf
Tool Manager: toolchain-xtensa-esp-elf@14.2.0+20241119 has been installed!
Verbose mode can be enabled via `-v, --verbose` option
CONFIGURATION: https://docs.platformio.org/page/boards/espressif32/esp-wrover-kit.html
PLATFORM: Espressif 32 (54.3.21) > Espressif ESP-WROVER-KIT
HARDWARE: ESP32 240MHz, 320KB RAM, 4MB Flash
DEBUG: Current (ftdi) On-board (ftdi) External (cmsis-dap, esp-bridge, esp-prog, iot-bus-jtag, jlink, minimodule, olimex-arm-usb-ocd, olimex-arm-usb-ocd-h, olimex-arm-usb-tiny-h, olimex-jtag-tiny, tumpa)
PACKAGES: 
 - framework-arduinoespressif32 @ 3.2.1 
 - framework-arduinoespressif32-libs @ 5.4.0+sha.858a988d6e 
 - tool-esptoolpy @ 5.0.0 
 - toolchain-xtensa-esp-elf @ 14.2.0+20241119
Converting WiFiScan.ino
LDF: Library Dependency Finder -> https://bit.ly/configure-pio-ldf
LDF Modes: Finder ~ chain, Compatibility ~ soft
Found 41 compatible libraries
Scanning dependencies...
Dependency Graph
|-- WiFi @ 3.2.1
Building in release mode
Compiling .pio/build/esp-wrover-kit/src/WiFiScan.ino.cpp.o
Retrieving maximum program size .pio/build/esp-wrover-kit/firmware.elf
Checking size .pio/build/esp-wrover-kit/firmware.elf
Advanced Memory Usage is available via "PlatformIO Home > Project Inspect"
RAM:   [=         ]  13.2% (used 43408 bytes from 327680 bytes)
Flash: [=======   ]  69.1% (used 905754 bytes from 1310720 bytes)
================================================= [SUCCESS] Took 14.54 seconds =================================================

Environment     Status    Duration
--------------  --------  ------------
esp-wrover-kit  SUCCESS   00:00:14.538
================================================= 1 succeeded in 00:00:14.538 =================================================
claudia@MacBook-Air-von-Claudia arduino-wifiscan % 

[maxgerhardt]

Maybe there were multiple version of the toolchain and you need to do rm -rf ~/.platformio/packages/toolchain-xtensa-esp-elf*? I reproduced this on Linux.

[maxgerhardt]

Extremely interesting: https://community.platformio.org/t/bb-spi-lcd-missing-esp-psram-h/48019/28?u=maxgerhardt

[Jason2866]

Whatever i delete in folder .platformio/packages it gets installed and compile is successful

[Jason2866]

Mhh, all CI examples are successfull. Testing on Ubuntu 24.04, Windows Server 2022 and MacOS 14 (arm) Github Runners.
Maybe local installations dont have all this packages?

I am developing on an MacBook Air (arm).

[Jason2866]

I wonder why there is no support for xz files? xz (lzma) is normally included in the used imported module tarfile
Maybe a left over from the time when support was switched off because the module lzma was hacked.

[Jason2866]

I strongly recommend to do Step 1 as explained in espressif IDF docu https://docs.espressif.com/projects/esp-idf/en/stable/esp32/get-started/linux-macos-setup.html#step-1-install-prerequisites
since nearly all tools from IDF are used from pioarduino Platform espressif32

[maxgerhardt]

Technically for Mac users this would not trigger the installation of xz right?

brew install cmake ninja dfu-util ccache
xcode-select --install
brew install python3

[Jason2866]

brew install python3 will install a full not crippled python with tar including xz (lzma) support. AND for brew install of XCode command line tools is needed which includes mandantory tools like git.

[Jason2866]

Shot in the dark. This modified version deletes folder with similar names.

platform = https://github.com/pioarduino/platform-espressif32.git#develop_pattern_remove

Which maybe left behind from old Platformio installations. Maybe this old folder do harm.

[Jason2866]

After a lot of tries i was able to reproduce a faulty behaviour. Using the branch develop_pattern_remove i was not able to produce any fail when switching between Platformio and pioarduino.
@maxgerhardt please try, if it solves for you too i will rerelease the pioarduino Platform release for Arduino core 3.2.1
and apply the changes from branch develop_pattern_remove in all branches

[Jason2866]

Tested with Ubuntu Linux 24.04 (arm) and MacOS. Both worked. For Ubuntu Linux it is needed to update the pre installed Python3 to the full version sudo apt install python3-full and to install xz support for tar with sudo apt install xz-utils