New lock file format

[wolfv]

It would be great to have a new lockfile format.
The current conda lockfile format (explicit env format) has quite a bunch of shortcomings: it's a weird ad-hoc format and only supports MD5 sums (and not even by default, I think & SHA256 is much better).
The command to export an explicit environment in conda is conda list --explicit [--md5]

Micromamba already improves on this by changing the command to micromamba env export --explicit [--no-md5] (ie. it uses the env subcommand and defaults to add --md5 hashes).

# This file may be used to create an environment using:
# $ conda create --name <env> --file <this file>
# platform: osx-arm64
@EXPLICIT
https://conda.anaconda.org/conda-forge/osx-arm64/argp-standalone-1.3-h3422bc3_0.tar.bz2#b744f29f1ef63fcedcb63b45d7ceed4a
https://conda.anaconda.org/conda-forge/osx-arm64/git-lfs-2.13.3-hce30654_0.tar.bz2#d0a6dda324b5d970c296dca838965193
...

I am thinking it would be nice to replace this with a proper YAML based format.

I am proposing something like:

metadata:
  spec: explicit-1.0
  description: ... # optional

name: myenv
# channels: ?
explicit-packages:
  linux-64:
  - name: xyz
    version: 0.15.0
    resolved: https://conda.anaconda.org/conda-forge/linux-64/xyz-0.15.0-had123.tar.bz2
    sha256: 123123123123123sjadalkjdlkajsk
    signature: ... ? # we need to also have certain metadata to validate signatures, though
  - name: pip
    version: 1.15.0
    resolved: https://conda.anaconda.org/conda-forge/noarch/pip-1.15.0-had123.tar.bz2
    sha256: 123123123123123sjadalkjdlkajsk
  osx-64:
  - name: xyz
    version: 0.15.0
    resolved: https://conda.anaconda.org/conda-forge/osx-64/xyz-0.15.0-had123.tar.bz2
    sha256: 123123123123123sjadalkjdlkajsk
  - name: abc
    version: 0.16.0
    resolved: https://conda.anaconda.org/conda-forge/osx-64/abc-0.16.0-had123.tar.bz2
    sha256: 123jk1lk23j1kl2j3kj12k3jlj1lk2

The explicit packages would contain a list per (supported) subdir. The list would be the full env resolution (including noarch pacakges) and in the correct order for installation (as current lockfiles today).

[mariusvniekerk]

This WIP PR conda/conda-lock#106 has a bunch of handy things that could also go into metadata

[mariusvniekerk]

This lockfile spec should have a version and ideally a reference to some standard jsonschema representation of the structure.

version: 1
$schema: https://some/url/for/schema_v1.json

[maresb]

Ooh, this is very exciting!!!

My thoughts regarding the lockfile metadata are that I'd like my lockfiles to be self-documenting. For instance, I'd like them to know how they were created, for example with which command. I'd also like to be able to add "comments" as explanation for colleagues. That way by looking at the lockfile, it'll be obvious what it is, where it came from, and how to update it.

I think it's useful to be able to choose which fields to include or exclude. For instance, some people may find it useful to include the timestamp and the username, but others might find the timestamp annoying with git, or might not want to leak their username.

I have also realized that the feature I'd really like is to be able to stick my "non-explicit" dependencies in the metadata so that I can run a command like conda-lock update environment.lock and have it rerun the solver and upgrade the lockfile in-place.

I was not bold enough to propose a new format, so I have been working with a header consisting of commented yaml. Obviously a proper YAML format like this would be better.

I have things mostly implemented; I'm just working on a system for versioning the metadata generation process so that it can be extended or modified in the future. I wanted to finish it this weekend but I ended up not having enough time.

[wolfv]

Yeah, we could make this format a super set of the existing YAML environment files. So that you could have

name: ...
metadata:
...
channels:
- conda-forge
- bioconda
dependencies:
- abc >0.5
- xyz =1.15
package-lock:
  linux-64:
     - ...

then some micromamba command could automatically update the lockfiles + certain metadata keys.

[maresb]

I think I already tested at some point whether or not conda would accept an extra top-level item in the environment file (like metadata), and unfortunately it didn't work. Thus we're unfortunately looking at a breaking change. (But who cares about conda anymore? 😉)

EDIT: Or was it mamba that didn't work??? Sorry, I take it back. I'm not sure anymore...

[wolfv]

hmm, I don't think micromamba would complain about extra keys. With mamba, we're just using conda code though, so it might happen, IDK! :)

[maresb]

For a normal environment file it seems to produce a warning.

$ conda env create -n testenv --file=env.yaml

EnvironmentSectionNotValid: The following section on '/env.yaml' is invalid and will be ignored:
 - metadata

For an explicit lockfile, I'm getting:

CondaValueError: invalid package specification: metadata: asdf

[wolfv]

Yeah, in explicit lockfiles you need to use a comment # metadata: whatever...

[mariusvniekerk]

@maresb yeah, none of this exists yet, we're designing what we want from a lockfile format for conda.

[maresb]

Here is a summary of what I came up with in my PR:

conda-lock-metadata:
  about: This lockfile was generated by conda-lock to ensure reproducibility.
  comment: |-
    Run the following command to update this project's dependencies.
  command: conda-lock -f environment.yml --metadata=all
  command_with_path: /root/conda/envs/conda-lock/bin/conda-lock -f environment.yml --metadata=all
  conda_lock_version: 0.11.3.dev0+gf2ba8d4.d20210904
  created_by: root
  input_hash: f15a045753a401da73dd7c1693fd031e0ad41c0b4c9ca8545c0a8ab56c21d16c
  platform: win-64
  timestamp: 2021-09-05 23:43:18+02:00
  metadata_version: v1
  dependencies:
    - mamba
    - conda-lock

On the command line, you should specify --metadata=v1,about,platform,command,dependencies or similar to select desired fields and specify their order.

[mariusvniekerk]

Thinking a bit through some of the human consumable parts for this would we want something like this instead

metadata:
  spec: explicit-1.0
  description: ... # optional
  channels:
    - conda-forge

name: myenv
packages:
  # probably will be alphabetically ordered
  xyz:
    linux-64:
      version: 0.15.0
      resolved: https://conda.anaconda.org/conda-forge/linux-64/xyz-0.15.0-had123.tar.bz2
      sha256: 123123123123123sjadalkjdlkajsk
      signature: ... ? # we need to also have certain metadata to validate signatures, though
    osx-64:
      version: 0.15.0
      resolved: https://conda.anaconda.org/conda-forge/osx-64/xyz-0.15.0-had123.tar.bz2
      sha256: 123123123123123sjadalkjdlkajsk
  pip:
    linux-64:
      version: 1.15.0
      resolved: https://conda.anaconda.org/conda-forge/noarch/pip-1.15.0-had123.tar.bz2
      sha256: 123123123123123sjadalkjdlkajsk
  abc:
    osx-64:
      version: 0.16.0
      resolved: https://conda.anaconda.org/conda-forge/osx-64/abc-0.16.0-had123.tar.bz2
      sha256: 123jk1lk23j1kl2j3kj12k3jlj1lk2
install_order:
  linux-64:
    - xyz
    - pip
  osx-64:
    - xyz
    - abc    

Noarch packages will still be repeated per platform since there may be a platform specific variant of something that is usually noarch.

By grouping the packages together we make it easier to review updates to lockfiles as when you relock you can ensure that all the versions you expect to move, move.

[wolfv]

I would also like to make it a superset of a regular yaml environment file, by the way.

[maartenbreddels]

you could also consider blake3 instead of sha256, I think it's much faster (parallel/multithreaded).

[wolfv]

I don't think the hashing speed matters so much here. One nice thing about sha256 is that we can directly pull from a OCI registry with it.