Improving remote test runner to cover new os/boards

[glistening]

Note: We will not cover system I/O automation in this issue so that we can concentrate remote-test-running itself

We are using remote test runner to run IoT.js unit tests automatically. It works greatly for NuttX/stm32f4-discovery and Linux/RPI.

Now, we must support two more platforms (TizenRT/artik053 and TizenIoT/RPI) and the kind of boards may grow. Therefore I would like to improve remote test runner to cover several board/os platforms with minimal cost.

We are thinking of using serial communication. Because

1. TizenRT does not support telnetd officially yet since it is under development. Serial communication is the only one that is verified and we can use safely. We don't need to enable and debug the features that the platform owner does not officially support.

2. We could use the serial communication for NuttX/stm32f4-discovery, Linux/RPI, etc..

@rtakacs Could you summarize the investigation of using serial communication here?

Any comments welcome.

[kolipka]

As a continuation of https://groups.io/g/iotjs-dev/message/21:

A user story:
Now I get test_spi.js failed. I need to know how many tests/asserts was executed for this module, and what tests failed. That will help me to estimate condition of module: 1 of 10 failed is differend that 9/10 failed. Now to know we have to read sources and extract this information. This is pretty basic capability of UT frameworks.

I disagree with "test-runner has no major weakness" because:

• way of storing results is flawed and have to be fixed ASAP - this is critical flaw. No multiple sources of data, if script fail and amend empty subset of files, all data will be lost( and part of webpage).
• whole runner will have to be refactored since it was designed to store small number of devices. Because of that OS/device tweaks are scattered in many places. This will be rather heavy since requirements as you point it above can't be meet using current approach
• we are making JS engine / nodejs like runtime. We are not using it, we do not learn from other similar products. How can we learn what is missing? We choose to use Python instead.
• remote-testrunner workaround issues we have in our test system in overall. eg.: test skiplists.

benefits from rewriting it in JS/nodejs as I see (iotjs in the future):

• promoting usage of JS
• learning about IoT.js ecosystem lacks
• the best in the world development ecosystem: nodejs + npm (vs python + pypi)
• https://www.npmjs.com/package/firebase maintained by google vs python-firebase last updated 3 years ago
• Since we target JS developers, we can expect contributions from JS developers

as I said - this is OK for now, as a prototype. I don't want to deminish effort that was done. We should use all sh scripts that configure and gathered knowledge at the first place. Our whole testing process should designed from scratch. I feel that we try to stich not fitting parts together in dirty way.

[rtakacs]

@glistening It seems feasible.

NuttX - STM32F4-Discovery: NuttX supports shell over USB (usbnsh), that's why we can replace the currently used Ethernet based communication (netnsh). We can use minicom for the serial communication. (Tested)

Linux - Raspberry Pi: We can use UART for the serial communication, so we can eliminate the Ethernet based one. minicom can be used as well for the serial communication. (Tested)

I can work on this process if that's OK for you.

[glistening]

@kolipka I don't agree that we need to rewrite whole test framework for the reason above. The requirement and benefit seems to be improvable in remote test runner, or lower priority than getting the working solution in given time schedule to meet basic requirements. ( Currently we have no way to run unit tests automatically for new platforms. Let's make working one before improving. )

I am not sure that JS/node solutions is superior to current solution. I would not choose nodejs and npm solution before convincing that the nodejs and npm solution will work well in iotjs environment, which has limited H/W resources (memory, flash) and limited S/W support (ES2015, Posix API, ...). We don't live in rich enivironment like Node.js.

I think @zherczeg's comment in mailing list is more reasonable and near to real. Please note that we don't asked you to design whole new test framework. I am afraid that we will not meet schedule in this way.

@rtakacs Yes, please work on.

[kolipka]

@glistening

1. I am not aware of any time schedle so I don't take it into account. What I want to emphasize is nodejs is superior to any python solution because of above reasons and many more. As a side note: I like python a lot, and spend few years using it exclusively and it was really good experience. But pleasae choose the best tool for the job. Python is not the best here.
   2a. I am not sure I get the second part. I want to have remote-testrunner in JS/Node. This works on desktop, not in constrained environment, the same way as python now.
   2b. JS developers, our customers, live in nodejs ecosystem. They expect having that high quality tools. Using tools like node, npm, wepback, babel, etc they target old constrained browsers/envs. Those tools are used only as dev environment helpers and are not embedded in a final product. Please compare package.json dependencies vs devDependencies section.
   2c. Existing JS test runners, eg.:Karma, are having quite reach plugin system. You can customize every aspect of it. Making some plugins would allow running tests on our constrained devices, and gathering results. I am pretty sure we don't need to reinvent the wheel. As @rtakacs said: "Before we start any rewrite, perhaps we should think about the effort / benefit ratio". Is there any benefit of making own test runner over already existing ones?
   2d. Our test suite does not answer basic question I expect that it should. Having simple API like describe, it, expect introduce low footprint, but provide a lot of info and quite nice structure to test suite.

I undestand that you want to have working solution, and I agree with this. My point is: think about your customers, what they expect.

[glistening]

@kolipka Your comment helped me much to understand your point in detail. I appreciate your suggestion.

[robertsipka]

@kolipka : Thanks for your suggestion to use firebase. It is good for our database logic because we don't want to handle complex queries and the data synchronization is quite good. I have changed the remote-testrunner and the related sites (Iot.js, Jerryscript) to store and retrieve results from a firebase realtime database.

[rtakacs]

@daeyeon @glistening @hs0225: @robertsipka and me had a discussion and we found a possible solution to use the remote testrunner in the IoT.js project. For now, the precommit.pyscript is able to create builds for all the targets (stm32f4dis, rpi2, artik053). This script should be extended with a flash mechanism. When passing the --flash, the precommit.py could download the appropriate flasher tool (into the deps folder), and use it for flashing.
After that, just the testrunner should be extended with an ssh and serial communication based on
the js-remote-test project.

With this approach, we could test our development on devices without downloading the js-remote-test project. On the other hand, the maintenance could be easier as well. The cost of the implementation is relatively low. Don't need to create a new architecture for the js-remote-test and several related code parts are already implemented in the js-remote-test. I hope we could have a prototype by the next week.

In JerryScript, we could do the similar things. We should have a precommit.py that could compile Jerry for the given targets. After that, the testrunner of JerryScript also should be extended with ssh and serial communication.

In this case both IoT.js and JerryScript could have an universal testrunner for all the targets.

[kolipka]

@rtakacs please remember about #1126. IMHO this script does too many things. The name is also missleading.

[rtakacs]

@kolipka Yes, you are right. The precommit.py should have a better name, or all the build methods should be moved to a target_builder.py file.

[LaszloLango]

@rtakacs sounds good.

[glistening]

@rtakacs I have similar opinion with @kolipka. precommit.py do too much things and is growing since it is introduced. My another concern is added as comment on #1212. Let's talk on IRC with @hs0225.