Create a testsuite for the standard

[certik]

Currently each compiler must develop its own tests for every feature in the standard. One could imagine, down the road, that the committee can maintain a "blessed" set of tests for each feature in the standard, which if the compiler passes, then the feature can be considered "implemented".

One can then maintain an automatic matrix of features and compilers to see which compilers implement which features.

Such a testsuite for each feature can be a nice complement to the standard, giving an example for all the corner cases how a certain feature should behave.

This might seem like a lot of work to do from scratch, but we can start doing it for every new feature from now on. And eventually implement the tests for old features as time allows. Doing it for new features would not be as hard, since the committee spends a lot of time designing each feature carefully. So writing tests for it might even make the process easier.

[FortranFan]

@certik this will be tremendously helpful.

Do you think GitHub can be used for this?

And do you think it will be possible to get a "critical mass" of support from the Fortran committee to engage and contribute to such an effort? For example, to review and confirm the standard-conformance of each of the tests in a suite because such a "blessing" is what will validate the test suite.

[certik]

@FortranFan Yes, GitHub or GitLab can be used. Regarding convincing the committee, I think as we keep discussing and proposing the various ideas in this repository, I think it will become clear in the future which ideas are very popular, and then we should all work towards convincing the committee to do those.

[klausler]

A thorough suite of positive and negative tests for every feature, requirement, and constraint in the standard would be incredibly valuable to implementations, as well as to commercial customers requiring proof of conformance.

[sblionel]

For FORTRAN 77, there was an official ANSI test suite which vendors would pay the US government annually to come in, run, and certify the results. The suite was rather simplistic overall and never changed. ANSI/NIST discontinued it.

There have been several attempts to create a more comprehensive test suite for newer revisions. The Hendrickson/Spackman SHAPE95 suite was licensed by multiple vendors, and it was good, but work stopped on it. While I was at Intel I became aware of an Italian user, whose name escapes me, who appeared to be creating a useful test suite covering all the syntax, but maybe not the full semantics. I have no idea what happened to that as we stopped hearing from him (bug reports) after a while.

This is a far bigger task than you might imagine, and requires serious, ongoing resources. It is not something the Fortran committee, all volunteers taking time from their day jobs, can develop or even specify/manage. As a former vendor, I can tell you that vendors would be happy to pay for a good test suite with ongoing maintenance and support. But who is going to create it? These vendors (and I'll include freeware developers in this) all have their own test suites, largely comprised of collected applications, unit and regression tests; but they are far from comprehensive, no matter how good intentioned.

Writing a good test is much harder than you might think. Compiler developers are the wrong people to write tests, as they will test what they think the feature is supposed to do. For a while, Intel had a dedicated team, separate from the compiler group, writing tests based on the documentation; it worked very well and uncovered many bugs, but the team, funded by a different organization, was disbanded and we lost that resource.

The big problem as I see it is that the market for such a test suite is small compared to the cost of creating and maintaining it. The only way I could see this happening is if some government funded the effort, as they did in the F77 days, and I don't see that happening today, especially for Fortran.

So, yes, it would be fantastic if a good, modern Fortran test suite existed and was maintained. Who is going to do it?

[certik]

@sblionel Thanks a lot for your feedback, I really appreciate it. My approach is to first figure out ideas that the community would like to see. This seems to be one of the more popular ideas.

Once we have established that we want to get this done, we can shift the discussion into how to get it done. I can see several approaches, and there might be more ways to get it done:

• Vendors can collaborate on a common opensource project. But as you said, in some sense they are the "wrong people" to write such tests.

• Vendors can all pool to fund a separate entity to write and maintain such a testsuite. As you said, vendors might be willing to pay for such a thing. There are several models how this can be done. I am at a NumFOCUS summit conference right now and there are many efforts to allow precisely such funding. One is OpenTeams, where the idea is that (opensource) projects can propose an idea, and OpenTeams facilitates the discussion with companies to sell the idea to them and companies can pool together to fund this. Writing modern Fortran testsuite might be a good fit. There are other ways to get it funded also.

So it might be possible to secure and facilitate the funding. The next question is who will do that. Again, there are several approaches here:

• I talked with @nncarlson a few days ago, and we are actually both interested in working on such a testsuite. But as you said, this is a much bigger task than two people can do. But we can help. This is just an example that it is possible to find interested parties to help.

• Compiler vendors can actually help a lot too, as long as this project is properly lead by somebody who understands the task at hand and can ensure the overall quality of the testsuite.

Somebody would need to assemble a team and lead this effort and try to get it funded. This comment can serve as some ideas how to do so.

[FortranFan]

@certik wrote:

.. My approach is to first figure out ideas that the community would like to see. ..

@certik, there might be some people like me who are restricted in terms of what they can do with GitHub (e.g., basic use via browser ok, but not much else) but who can contribute some unit tests for consideration toward inclusion in the "modern Fortran testsuite".

If it's possible for you or others on such a team on the "standard" testsuite to set up some kind of infrastructure for such a suite (perhaps on GitHub itself?) and some mechanism for community submissions which can then be reviewed for correctness/suitability, etc. particularly in terms of standard conformance (or lack thereof as intended failures) and approved for inclusion in the suite, that might be another resource this team can start drawing upon.

Over the years, I've assembled quite a few "informal" and small unit tests on a variety of standard features including OO facilities, parameterized derived types, UDDTIO, coarrays, interoperability with C, pure and elemental subprograms, submodules, block constructs, etc. which I can start to "clean up" and share with the team working on the "standard testsuite". And there might be other community members willing to contribute similarly.

To show an example, here's a case I've been grappling with the last couple of weeks: I think the code is conforming per my read of current standard, however it fails with 2 processors I tried which then makes me wonder if the compilers have bugs in their implementations or whether I'm in the wrong. If it is the former i.e., compiler bugs, then this might be a possible case for inclusion in the "modern Fortran testsuite":

#include <stdio.h>
#include <assert.h>
#include "ISO_Fortran_binding.h"

void Csub(const CFI_cdesc_t *, size_t);

void Csub(const CFI_cdesc_t * dv, size_t locd) {

   CFI_index_t lb[1];
   lb[0] = dv->dim[0].lower_bound;
   size_t ld = (size_t)CFI_address(dv, lb);

   printf("In C function: CFI_address of dv = %lx\n", ld);
   assert( ld == locd );
   return;

}

! Unit Test #: Test-1.F2018-2.7.5
! Author     : FortranFan
! Reference  : The New Features of Fortran 2018, John Reid, August 2, 2018
!              ISO/IEC JTC1/SC22/WG5 N2161
! Description:
! Test item 2.7.5 Fortran subscripting
! void *CFI_address(const CFI_cdesc_t *dv, const CFI_index_t subscripts[]);
! that returns the C address of a scalar or of an element of an array using
! Fortran sub-scripting.
!

   use, intrinsic :: iso_c_binding, only: c_int, c_size_t, c_loc

   implicit none

   integer, parameter :: LB_A = -2
   integer, parameter :: UB_A = 1
   character(len=*), parameter :: fmtg = "(*(g0,1x))"
   character(len=*), parameter :: fmth = "(g0,1x,z0)"

   blk1: block

      interface
         subroutine Csub(a, loc_a_1) bind(C, name="Csub")
            import :: c_size_t
            type(*), intent(in) :: a(:)
            integer(c_size_t), intent(in), value :: loc_a_1
         end subroutine
      end interface

      integer(c_int), target :: a( LB_A:UB_A )
      integer(c_size_t) :: loc_a

      print fmtg, "Block 1"

      loc_a = transfer( c_loc(a(lbound(a,dim=1))), mold=loc_a )
      print fmth, "Address of a: ", loc_a

      call Csub(a, loc_a)

      print *

   end block blk1

   blk2: block

      interface
         subroutine Csub(a, loc_a_1) bind(C, name="Csub")
            import :: c_int, c_size_t
            integer(kind=c_int), allocatable, intent(in) :: a(:)
            integer(c_size_t), intent(in), value :: loc_a_1
         end subroutine
      end interface

      integer(c_int), allocatable, target :: a(:)
      integer(c_size_t) :: loc_a

      print fmtg, "Block 2"

      allocate( a( LB_A:UB_A ) )
      loc_a = transfer( c_loc(a(lbound(a,dim=1))), mold=loc_a )
      print fmth, "Address of a: ", loc_a

      call Csub(a, loc_a)

      print *

   end block blk2

end

Here're the compilation and linking steps used using MinGW gfortran along with program execution which shows the failure:

C:\Temp>type c.c
#include <stdio.h>
#include <assert.h>
#include "ISO_Fortran_binding.h"

void Csub(const CFI_cdesc_t *, size_t);

void Csub(const CFI_cdesc_t * dv, size_t locd) {

   CFI_index_t lb[1];
   lb[0] = dv->dim[0].lower_bound;
   size_t ld = (size_t)CFI_address(dv, lb);

   printf("In C function: CFI_address of dv = %I64x\n", ld);
   assert( ld == locd );
   return;

}

C:\Temp>x86_64-w64-mingw32-gfortran.exe -c -Wall -Wextra c.c -o c.o

C:\Temp>type p.f90
! Unit Test #: Test-1.F2018-2.7.5
! Author     : FortranFan
! Reference  : The New Features of Fortran 2018, John Reid, August 2, 2018
!              ISO/IEC JTC1/SC22/WG5 N2161
! Description:
! Test item 2.7.5 Fortran subscripting
! void *CFI_address(const CFI_cdesc_t *dv, const CFI_index_t subscripts[]);
! that returns the C address of a scalar or of an element of an array using
! Fortran sub-scripting.
!

   use, intrinsic :: iso_c_binding, only: c_int, c_size_t, c_loc

   implicit none

   integer, parameter :: LB_A = -2
   integer, parameter :: UB_A = 1
   character(len=*), parameter :: fmtg = "(*(g0,1x))"
   character(len=*), parameter :: fmth = "(g0,1x,z0)"

   blk1: block

      interface
         subroutine Csub(a, loc_a_1) bind(C, name="Csub")
            import :: c_size_t
            type(*), intent(in) :: a(:)
            integer(c_size_t), intent(in), value :: loc_a_1
         end subroutine
      end interface

      integer(c_int), target :: a( LB_A:UB_A )
      integer(c_size_t) :: loc_a

      print fmtg, "Block 1"

      loc_a = transfer( c_loc(a(lbound(a,dim=1))), mold=loc_a )
      print fmth, "Address of a: ", loc_a

      call Csub(a, loc_a)

      print *

   end block blk1

   blk2: block

      interface
         subroutine Csub(a, loc_a_1) bind(C, name="Csub")
            import :: c_int, c_size_t
            integer(kind=c_int), allocatable, intent(in) :: a(:)
            integer(c_size_t), intent(in), value :: loc_a_1
         end subroutine
      end interface

      integer(c_int), allocatable, target :: a(:)
      integer(c_size_t) :: loc_a

      print fmtg, "Block 2"

      allocate( a( LB_A:UB_A ) )
      loc_a = transfer( c_loc(a(lbound(a,dim=1))), mold=loc_a )
      print fmth, "Address of a: ", loc_a

      call Csub(a, loc_a)

      print *

   end block blk2

end

C:\Temp>x86_64-w64-mingw32-gfortran.exe -c -std=f2018 -Wall -Wextra p.f90 -o p.o

C:\Temp>x86_64-w64-mingw32-gfortran.exe c.o p.o -o p.exe

C:\Temp>p.exe
Block 1
Address of a:  87FE10
In C function: CFI_address of dv = 87fe10

Block 2
Address of a:  A3930
In C function: CFI_address of dv = a3928
A s s e r t i o n   f a i l e d !

 P r o g r a m :   C : \ T e m p \ p . e x e
 F i l e :   c . c ,   L i n e   1 4

 E x p r e s s i o n :   l d   = =   l o c d

This application has requested the Runtime to terminate it in an unusual way.
Please contact the application's support team for more information.

Program received signal SIGABRT: Process abort signal.

Backtrace for this error:
#0  0xffffffff
#1  0xffffffff
#2  0xffffffff
#3  0xffffffff
#4  0xffffffff

C:\Temp>

[certik]

@FortranFan I will setup the infrastructure very soon and I will update this issue when I do so.

Here is my plan so far:

The testsuite would be separate and independent of any compiler. It would contain metadata about what feature is being tested, which files, and any other useful info, and then it would also contain (probably Python) library showing how to load all the tests and process them.

There can be lots of backends, either as part of the test suite, or separate, for things like autogenerated CMake build system to test any compiler, or to generate a nice website with the coverage for each compiler, etc.

Then compilers, such as LFortran, can have scripts that take this and process the files and create an LFortran specific testsuite from it, for example I'd like to generate parts of this test file.

Other compilers could do something similar. For example for Flang the files in https://github.com/flang-compiler/f18/tree/fdb351ca2afb0d71028785da4687113343e11f54/test/semantics could be generated from such a testsuite. One downside is that the Flang testsuite has a compiler dependent checks for error messages, so it might not be possible to generate such tests from a compiler independent test suite.

One worry that I have is that if the testsuite files change (for whatever reason), then when Flang or LFortran specific testsuite gets regenerated, it will break (the test won't pass anymore). Or to formulate it in another way --- to what extent can a compiler testsuite be "outsourced" to a compiler independent testsuite? Can it even work? I don't have the answer.

If it cannot be done, then the answer is that each compiler still has to maintain its own testsuite, and the compiler independent "standard" testsuite is then used to automatically check what features are being implemented by each compiler, and also each compiler can at least take the standard testsuite as a starting point and then adapt it to its needs.

@klausler what are your thoughts on this?

[klausler]

I think that the tests themselves are what's important, not the test suite infrastructure around them, so I would avoid getting too tightly integrated with tools like Python that change incompatibly over the years. Test cases that can be run manually through a compiler without knowledge of infrastructure are the most useful.

In practical terms, I suggest making a distinction between three kinds of tests:

1. Tests meant to compile and execute successfully if a particular requirement/constraint/feature is correctly and completely implemented.
2. Tests meant to compile successfully but abort during execution as they violate some constraint or "shall" clause; execution to completion means that the implementation has failed the test.
3. Tests meant to not compile successfully, with errors being indicated at/near some locations in their source code.

Tests in the first group should be as self-contained as possible and execute in such a way that it's easy to determine from the shell or other tools that they terminated happily; perhaps via STOP 'PASS'.

Tests in the second group should terminate with an expected error code at runtime; if they run to completion, they should indicate to the shell or other tools that they failed to detect a required error, perhaps via STOP 'FAIL'.

Tests in the third group should have their source code marked with comments describing expected error messages. Testing this group against a particular compiler may be best done by capturing error messages from that compiler, validating them manually, and saving them as "known good" compiler output, deviation from which may indicate errors later. If a compiler fails to catch an error and the program somehow survives to execution, it should crash with a message, perhaps via ERROR STOP 'FAIL'. It would be best if tests in this third group were as small as possible, eliciting a single error message, so that useful results could be obtained just by throwing these cases at a compiler and verifying that it refuses to compile each of them.

[certik]

@klausler thanks for the feedback. What would be some example in the category 2.? Things like sqrt(-1._dp) where the standard says "its value shall be greater than or equal to zero"? That's a great point.

What's your opinion on whether compilers like Flang could (in principle) use such a testsuite, or whether compilers will have to maintain their own testsuite anyway.

[klausler]

It's not either/or. I would love to have access to an authoritative suite of tests for the current revision of the standard, so that I wouldn't have to wade my way through all the ambiguity and imprecision in the text itself in an attempt to write tests for it. An authoritative test suite would reduce the size of f18's future test directories as the standard evolves; and if the committee had to develop or subcontract the development of such tests, perhaps ambiguity would be exposed earlier and fixed sooner. As is, we have access to many open- and closed-source test suites as well as applications, and depend heavily on all of them.

Oh, one last point that's really more important than it might seem: the Fortran standard updates section, requirement, and constraint numbers in incompatible ways with each revision. It's important for f18 compiler and test source code to refer frequently to the standard document. When Fortran 202x arrives, all of those textual citations are going to have to be updated if the standard continues to gratuitously renumber them. The C++ language standard has moved to using names rather than numbers for these referential purposes.

[FortranFan]

@klausler wrote:

I think that the tests themselves are what's important, not the test suite infrastructure around them

I would like to make it abundantly clear when I used the phrase "set up some kind of infrastructure for" a test suite, I only meant something that can help community members submit cases - like the example I showed upthread - for consideration and which can then be reviewed by the team for suitability. It'll be useful to some (basic) indexing scheme to reference the tests (perhaps something that follows a numbering scheme of features in the standard might help?) and preferably some description/comments/results summary to go along with these tests.

[klausler]

Re: numbering of clauses, constraints, and requirements:

In Principles and rules for the structure and drafting of ISO and IEC documents

section 5.6 (page 8):

Consistency should be maintained within each document, and within a series of associated
documents.
• The structure of associated documents and the numbering of their clauses should, as far as
possible, be identical.

If the series of Fortran standards constitute a "series of associated documents", future revisions should avoid needless renumberings of their contents.

[sblionel]

Again, I'll note that the numbering is largely due to ISO rules for standards documents. These are why the chapter numbers changed by three this time around and notes don't have section numbers. Syntax rules and constraints have to be sequentially numbered and there's no feasible way to keep the numbering consistent without freezing the language, as both get additions and deletions over time. Clauses (chapters) are consistent across versions, subject to ISO constraints, and added clauses (for example, C interoperability.)

I have sympathy for code and documentation (including error messages) that want to refer to specific sections of the standard, but I just don't see a way to get there other than qualifying these references with a specific standard and updating them once the implementation fully supports a revision. The DEC/Compaq/Intel compiler had many such references in error messages, but we ended up taking them out.