Parallelize the construction of constraints & co. of abstract models

[larsschellhas]

Right now a huge amount of the time used for instantiating an abstract model goes into evaluation and construction of constraints.
Those constraints are all dependent on the parameters they were given, but don't depend on each other. Therefore it should be possible to generate the expressions in parallel and combine them afterwards to the full model.
This would probably speed up the model instantiation extremely, especially on machines with many cores.

[mrmundt]

In an effort to clean up old issues, we are moving some design discussions to our Archived Design Discussions wiki.

[DLWoodruff]

Parallel construction of constraints would be a huge win, particularly for modelers who were aware of the possibility. Is there some reason this cannot be done for ConcreteModels?

[jsiirola]

The problem is the Python GIL: what we would really like is multithreaded model construction, but the GIL prevents true threading in Python. Some options we have considered include:

• wait for GIL-less Python
• Construct the model in a multi-process parallel environment. We would then need to either:
  • pickle the model parts and bring them back to the main process (the un-pickling process is just about as slow as creating the original model, and has to be done serially due to the GIL)
  • leave the model distributed and develop a distributed (parallel) solver interface (e.g., a parallel LP writer). Probably doable, but really time-consuming
• build the model in parallel in a C extension where threading is allowed. This could potentially be developed through something like APPSI,, but complicates the model representation, as the model is no longer pure Python and instead lives (mostly) on the C side

The option we are currently exploring [Jan 2024] is to leverage expression templates to avoid the construction of the individual constraint expressions. The trick will be updating the solver interfaces (LP / NL writers) to avoid expanding the constraints as part of the model compilation process.