Summary of my Google Summer of Code
Adding a configure Command
This first task gave me the occasion to study the distutils2 codebase. I saw a lot of good ideas in 4Suite’s DistExt modules and took some inspiration from their config command. At the end of GSoC, the configure command is written, tested and has basic documentation. This command accepts every option that build and install do (failing if there are conflicts in the install options, like install itself does), and writes them in a cache file. The build and install command take options from this file, which saves typing. It’s also possible to run configure build install in the same command line (the internal behavior will get cleaner thanks to some changes I have in mind; Alexis keeps telling me I should document the four or five successive designs I followed for this command.). There is still a nasty bug to fix.
An additional goal is to include the cache file created by the configure command in source or built distributions, so that for example plugins can get information about the build of the project they extend. It would be something similar to the sysconfig module used to get information about the compilation of the Python interpreter, but for Python projects. It would also allow using distutils to install but something else to build, as long as the cache format contains enough information for both parties. (This feature was not implemented during GSoC, for lack of a written description and lack of time.)
- Bug report
- Code (branch named “configure”)
- Documentation
Moving Setup Information to a Declarative Format
This second task was as motivating as the first since it shared the goal to make users’ and developers’ lives easier. The first step was to make it possible to describe the metadata in setup.cfg instead of setup.py. The proof of concept was ridiculously easy to write, thanks to the nice distutils2.dist module. For the second step, replacing setup(modules=…, scripts=…) by new sections and fields in setup.cfg, we missed a specification, so I took some time to write a design document describing it all. I have received useful feedback, and peer review is still ongoing.
As is now the rule in distutils2, I’ve started refactoring useful functions into a new module, which will be used by our higher-level code and can also be useful for third-party programs. Those support functions that are not yet written are mostly string operations and a bit of I/O, so it’s not hard but still important enough to deserve thorough tests and good documentation.
With the implementation of those new config file sections, setup scripts will effectively be obsolete. A new module or script shipped with distutils2 (therefore in the standard library when we merge back) will be the new way to run commands. This script is a two-liner that I haven’t committed anywhere yet since it’s, well, two lines.
The resources section was out of scope for this GSoC, since Tarek had said it would require a new PEP. I will volunteer to write the sample implementation required by the PEP when time comes to start it.
- Bug report for metadata
- Bug report for files
- Code (branch named “killsetup”)
- Design document
Looking Back
In those twelve weeks, I have learned a great deal about distutils2’s internals, most importantly the commands and Distribution class, but also the standalone modules like version or metadata. Looking into 4Suite’s DistExt was also useful, there are gems there that we should take, polish and include. Reading other students’ changesets helped me learn about things that I don’t really know, like dependency graphs or networking. Teamwork was also an opportunity to learn tools like code review sites, use email effectively or dive in the arcane of merge tools.
I really enjoyed test-driven coding, either with unit tests or “run and look at the output” manual tests. The big refactor I did to move code from the build and install commands to configure would not have been possible without all the unit tests for those commands to make me confident. Python is also excellent since it’s natural to write a docstring after the function declaration, so you write doc before the code without thinking about it or having to make it a religion. Last, I enjoyed doing lots of miscellaneous fixes, cleanups and improvements throughout the code base. Fixing pep8 or pyflakes warnings is easy and gives me an occasion to have a glance at modules I don’t know.
Unplanned Tasks
Some time before midterm, Tarek gave push rights to his main repository to Alexis and me, to have more manpower for reviews and pulls. I found myself in the position of repository gatekeeper, answering to review requests, replying to nearly each thread in the mailing list, and pulling from other students’ clones when I judged something ready. It was strange at first to have this power and duty, but I grew confident, I tried to be responsive and fair, and it seems I did a good job. I particularly liked sprinting remotely with the nice people from Montreal Python, being able to answer their questions, and rewarding their volunteer work with a push into the main repo right at the end of the sprint.
Synchronizing the documentation was not the easiest task. In the standard library, some improvements to the documentation of distutils had been removed because of the feature revert and freeze. Ali Aafshar did a bit of Subversion exploring, I completed his work with some missing bits, and we had a starting point. Alexis and I had chatted about the many goals of distutils2 and made plans to reorganize some parts of the docs to address the needs of users wanting to install something, developers wanting to distribute their project, and packaging developers wanting to reuse parts of distutils2 or extend it. Some moves have taken place, there is still a lot to do, but a solid foundation is here.
This extra work I did explains why my mentor was satisfied with me before the final deadline even though my tasks were not finished.
The Future
We have work ahead of us, new features to add, documentation to improve, and I am staying in the project as a volunteer. Reaching out to the Python communities and advocating distutils2 won’t be our easiest task, so the more people the better. While working on distutils2, I’ve also been following Python’s bug tracker, doing triage and reviews, and reading a number of Python mailing lists, with the occasional reply. The feeling of being a part of the Python community and being able to give back is warm and fuzzy, especially when you’re given acknowledgment and trust.
I’m still awed and thrilled to be a Python core developer. I have started making patches to fix bugs in distutils, freeing Tarek to focus on distutils2. Remote teamwork was good, and now that I’ve met Alexis and other fellow Pythonistas in real life and had such good times, I know that even though the university will have to get priority this year, I will stay involved.
Thanks
First, I want to thank Google for this incredible experience we’ve lived. Their Open Source department and especially Carol Smith are doing a great job and deserve warm applause.
I am also deeply grateful to the Python Software Foundation. This summer has been amazing, and I’m very happy to be part of a project with such excellent people, to work on such an excellent programming language.
Tarek Ziadé has been a terrific over-mentor. He took the time to phone me to answer a ton of questions, he helped us keep on track with the weekly meetings, he gave trust and praise. Thank you for everything.
C. Titus Brown is modest and says that he’s not done much to mentor me. Facts don’t matter; I needed guidance more than once, and he was here to give it. Sincere thanks to you.
Many people have supported me: Shashwat Anand, without whom I would not have applied at all, roommates, friends and family. The job has not been easy each week, but it was great to be given a chance even without formal computer science or engineering background, and to get evaluations based on work. It also meant something at a personal level; I’ll talk about that in another blog post.
I’ve saved the place of honor for the other distutils2 students: Alexis, Josip, Konryd, Zubin, Mouad (not in the distutils2 group but keeping in touch weekly). I’ve found the atmosphere in our group (students, volunteers, mentors, experts) excellent, without personal conflicts or flamewars. We worked and had fun, we helped each other, we cared about each other. Sharing this experience with you all was a pleasure and an honor.
