Skip to main content

2016-06-03

Time: 14:00 UTC

Hangout link: https://hangouts.google.com/call/v5olhwzpfzgzpoq5i3wthjpqpie

Attendees

Ray, Matt, Jonathan, Phil, Jonas, Michael, Philippe, John, Bjorn Gruning, Jan

Standing items

  • How many repos?
  • How many contributors?
  • New core devs?

Agenda

  • PyPI metadata redundancy

  • Python3 vs Python==3

  • How to depend (inc build depend) on applications which require Python 3, from a Python==2 env

  • 'Subenvironment dependencies' are a possible alternative

  • adding soname implies cohabitation. This is not always possible. Add soname in these cases?

  • bootstrapping: sometimes an older dependency is needed to build a current thing (circular dependencies may require subenvironments also)

  • Conda build to get split builds

    *   runtime packages will have sonames
    • dev packages will not - they will have versions. This enforce mutual exclusivity. Given version of dev package then appropriately determines runtime dependency soname.

  • Subenvironments hackathon proposed at SciPy 2016 (July 11-17)

  • Low level packaging

  • NetCDF (also curl/ca-certificates and Perl packages) - Done?

    *   curl and ca-certificates are done and available.

    *

    • Perl is no longer relevant as part of this process

  • GitHub rate limiting. How can we further mitigate these?

  • Add namespace to packages `node-`, `ruby-`, `perl-`, `why not python-` ;-)
    • 'Practicality beats purity' ;-)
    • At least at first, but i don't find this generally true.
    • One of the things proposed at continuum is the notion of primary namespaces - ones that effectively defined a default prefix of the namespaced for the package. This might be the best of both worlds. You could have ordered priority, too: search python-* first, then node-* next, then finally the full package name with no prefix. This priority would be defined by per-environment condarc perhaps, with initial saying depending on what packages get installed. For example, creating an env with python installed first would make python primary env.
    • I can understand the attraction of that, but it seems like a potential source of considerable confusion (e.g. why does installing x work differently in this environment to that one?). Maybe this would be more workable if namespaces were actually part of a new syntax, rather than just prefixes on package names.
    • Sure, that's reasonable - have the namespace search thing be a user-defined convenience thing, rather than an automatically determined thing.
    • It is worth keeping in mind that the Python naming change would be a big break from existing Continuum packages. So, this decision should not be taken lightly.

  • Another thing to consider here might be a new piece of metadata. For instance, we could specify the primary language of a package. We could then specify to conda install that we want this language of a package. Possible syntax might include something that looks like that of the above. Not sure how we want to handled conflicts if we want to error, warn and install everything, or something else.

  • A simpler idea that we might consider that includes some of the ideas Michael mentioned above, but could be implemented without changes to conda or package metadata would be to place packages in labeled channels. That way all Python packages would be in conda-forge/label/python. This way one could simply add this labeled channel and get all the python packages one wants. It's still a little fragile when enabling multiple labels, but maybe this can leverage the channel resolution stuff that Michael Grant has worked on.

  • PR reviews

    *   Treat every PR as a Work in Progress. At least let PRs sit for a few hours before merging them.
    • Wait for answers when we ask clarification questions and avoid acting before we have them.
    • Respect the first reviewer by not repeating her/his review comments with another words. That is also bad for the person submitting the PR as it is confusing.
    • Avoid the death by a thousand cuts: Many small "nit" comments that might scare new contributors ( ping Mike S ;-)

  • Notifications (how do we stay on top of them)

  • More compiler fun:

  • MSYS2

    *   Discussing Ray Donnelly's work on MSYS2 packages and how we want to use and integrate these into conda-forge.
    • Some use cases to consider OpenBLAS, FFTW, build tools, others?

  • Binary data

    *   Do we include it in recipes?
    • What kinds do we allow if any (e.g. icons)?
    • How do we verify the licensing?
    • How do we verify that they are safe?

  • OpenBLAS (on Windows)

  • Dev releases: Where do they happen?

    *   Do we do them at conda-forge?

            *   Maybe add a label.

    *   Do we let others do them with a feedstock on their own repo?
    • How do we enforce whatever we decide?

  • Conda-forge installer

    *   We have Python 3.5 now
    • Still need conda.
    • New repo?
    • Where do we host the installers? Git tags?

  • GitHub rate limitations. How can we further mitigate these?

    *   GitHub letter ( [](https://docs.google.com/document/d/19HLtYPwg6IKAwmxPwL7Vd3AX0n47ANP-ZTpZROn-Cwc/edit?pref=2&pli=1)[https://docs.google.com/document/d/19HLtYPwg6IKAwmxPwL7Vd3AX0n47ANP-ZTpZROn-Cwc/edit?pref=2&pli=1](https://docs.google.com/document/d/19HLtYPwg6IKAwmxPwL7Vd3AX0n47ANP-ZTpZROn-Cwc/edit?pref=2&pli=1) ).

  • Channel mirroring.

  • Feedstock history

    *   Is it sacred?

    • Do we rebase/force push?

          *   If so, under what conditions?
      • How do we avoid multiple people doing this simultaneously?

  • Consider applying to be a Numfocus sponsored project.

  • name native lib packages after SONAME -> conda forge/conda forge.github.io#157

  • Continuum metadata request: can we add these to linter?

    *   example metadata: [](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)[https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44](https://github.com/ContinuumIO/anaconda-recipes/blob/master/anaconda-build/meta.yaml#L36-L44)
    • Also, distinguish summary (limit of 77 or 80 chars) from description (unlimited)

  • Google hangouts has a max capacity of 10. Is it worth considering other methods of communication so everyone who wants to participate can?

Notes

3 weeks since last meeting

587 repos, 105 contributors (but some bots)

Suggestion that Patrick Snape be added as a core dev

PyPI metadata redundancy

Jinja template may be suitable to fill in this data from PyPI metadata

Related to question on how to maintain conda packages for pure Python packages, suggest to use existing feedstock setup. Seems everyone present agrees on this.

PyPI RSS/Twitter to check for new versions

Atom feeds of GitHub of releases

Naming library packages by soname

libpng16/17, pinning must be updated and recompiled can cause issues.

Suggestions to change packages names to sonames (libpng16, libpng17, ...) then multiple versions change

What about headers, they are un-versioned.

Can we install multiple versions of the same library in a single environments?

Split dev package (with headers) from libraries?

Can we track headers by version numbers?

What happens when we load multiple versions of a library into memory, does symbol resolution work? -- possibly no

Shadowing system libraries can cause issues

devel packages would be mutually exclusive, versioned

library packages named by soname

Need to be sure that two versions of same libraries headers cannot be brought into the same build environment which would cause issues

conda build needs to support split packages, good test cases

  • Discussion about splitting packages: conda/conda#793#issuecomment-174446435

Decisions:

  • Add soname to runtime packages
  • dev packages will be versioned but not include sonames
  • Task: Jan will write down proposal for libpng soname naming -> conda forge/libpng feedstock#7
  • Task: split packages in conda-build, open issue in repo

Python 3 vs python==3

"sub-environments", to allow for access to Python 2 and 3 in same environment.

Do we want to be able to have multiple runtimes in same enviroment

Do not really want to do this, conda environments are cheap

sub-environments have been needed for boot-strapping self-hosting compilers. Perhaps discuss/work on this at SciPy

Association with NumFocus

Requires three members without shared affiliation

Could get non-profit status

Funding for larger/longer build services

Qt build and other long builds

Can also Travis/other to have longer build times

Would be nice to have some of our own servers

Rackspace works with NumFocus and provides free VM times

Asking broader community for help, servers, package hosting, etc

Adding namespaces to packages

Should this be a requirements?

Prefix with language

Folders?

How about numpy, should it be python-numpy

How about when installing?

  • conda install python-numpy python-scipy?

Would require a change in conda

Warning

Prefix all non-python packages

Dependency only packages, pandas depends on python-pandas

GCC

Should recipes be annotated with compilers and version

gcc package which only checks the version

gcc dev-packages are really magic

conda-forge docker image ( https://github.com/conda-forge/docker-images/tree/fbde090bd608caa720d5caad861aa382a8bf3f5c/linux-anvil )

Special meeting to discuss compilers (MSYS2 too?)

  • 14:00 UTC next Thursday (Thursday June 9)
  • Look at each others docker images

Next general meeting three weeks from now

  • 14:00 UTC (Friday June 24th)

SciPy, BOFs, Sprints, Lighting talk on first day

  • I would like to prepare a quick intro "how to conda-forge" showing the work-flow from staged-recipes to updating a feedstock. Either in the both or as another lightning talk. (Preferably after Jonathan's LT.)