Final Report: A Distutils2 Summer of Code

Summary of my Google Summer of Code

Adding a configure Command

This first task gave me the occasion to study the distutils2 codebase. I saw a lot of good ideas in 4Suite’s DistExt modules and took some inspiration from their config command. At the end of GSoC, the configure command is written, tested and has basic documentation. This command accepts every option that build and install do (failing if there are conflicts in the install options, like install itself does), and writes them in a cache file. The build and install command take options from this file, which saves typing. It’s also possible to run configure build install in the same command line (the internal behavior will get cleaner thanks to some changes I have in mind; Alexis keeps telling me I should document the four or five successive designs I followed for this command.). There is still a nasty bug to fix.

An additional goal is to include the cache file created by the configure command in source or built distributions, so that for example plugins can get information about the build of the project they extend. It would be something similar to the sysconfig module used to get information about the compilation of the Python interpreter, but for Python projects. It would also allow using distutils to install but something else to build, as long as the cache format contains enough information for both parties. (This feature was not implemented during GSoC, for lack of a written description and lack of time.)

Moving Setup Information to a Declarative Format

This second task was as motivating as the first since it shared the goal to make users’ and developers’ lives easier. The first step was to make it possible to describe the metadata in setup.cfg instead of setup.py. The proof of concept was ridiculously easy to write, thanks to the nice distutils2.dist module. For the second step, replacing setup(modules=…, scripts=…) by new sections and fields in setup.cfg, we missed a specification, so I took some time to write a design document describing it all. I have received useful feedback, and peer review is still ongoing.

As is now the rule in distutils2, I’ve started refactoring useful functions into a new module, which will be used by our higher-level code and can also be useful for third-party programs. Those support functions that are not yet written are mostly string operations and a bit of I/O, so it’s not hard but still important enough to deserve thorough tests and good documentation.

With the implementation of those new config file sections, setup scripts will effectively be obsolete. A new module or script shipped with distutils2 (therefore in the standard library when we merge back) will be the new way to run commands. This script is a two-liner that I haven’t committed anywhere yet since it’s, well, two lines.

The resources section was out of scope for this GSoC, since Tarek had said it would require a new PEP. I will volunteer to write the sample implementation required by the PEP when time comes to start it.

Looking Back

In those twelve weeks, I have learned a great deal about distutils2’s internals, most importantly the commands and Distribution class, but also the standalone modules like version or metadata. Looking into 4Suite’s DistExt was also useful, there are gems there that we should take, polish and include. Reading other students’ changesets helped me learn about things that I don’t really know, like dependency graphs or networking. Teamwork was also an opportunity to learn tools like code review sites, use email effectively or dive in the arcane of merge tools.

I really enjoyed test-driven coding, either with unit tests or “run and look at the output” manual tests. The big refactor I did to move code from the build and install commands to configure would not have been possible without all the unit tests for those commands to make me confident. Python is also excellent since it’s natural to write a docstring after the function declaration, so you write doc before the code without thinking about it or having to make it a religion. Last, I enjoyed doing lots of miscellaneous fixes, cleanups and improvements throughout the code base. Fixing pep8 or pyflakes warnings is easy and gives me an occasion to have a glance at modules I don’t know.

Unplanned Tasks

Some time before midterm, Tarek gave push rights to his main repository to Alexis and me, to have more manpower for reviews and pulls. I found myself in the position of repository gatekeeper, answering to review requests, replying to nearly each thread in the mailing list, and pulling from other students’ clones when I judged something ready. It was strange at first to have this power and duty, but I grew confident, I tried to be responsive and fair, and it seems I did a good job. I particularly liked sprinting remotely with the nice people from Montreal Python, being able to answer their questions, and rewarding their volunteer work with a push into the main repo right at the end of the sprint.

Synchronizing the documentation was not the easiest task. In the standard library, some improvements to the documentation of distutils had been removed because of the feature revert and freeze. Ali Aafshar did a bit of Subversion exploring, I completed his work with some missing bits, and we had a starting point. Alexis and I had chatted about the many goals of distutils2 and made plans to reorganize some parts of the docs to address the needs of users wanting to install something, developers wanting to distribute their project, and packaging developers wanting to reuse parts of distutils2 or extend it. Some moves have taken place, there is still a lot to do, but a solid foundation is here.

This extra work I did explains why my mentor was satisfied with me before the final deadline even though my tasks were not finished.

The Future

We have work ahead of us, new features to add, documentation to improve, and I am staying in the project as a volunteer. Reaching out to the Python communities and advocating distutils2 won’t be our easiest task, so the more people the better. While working on distutils2, I’ve also been following Python’s bug tracker, doing triage and reviews, and reading a number of Python mailing lists, with the occasional reply. The feeling of being a part of the Python community and being able to give back is warm and fuzzy, especially when you’re given acknowledgment and trust.

I’m still awed and thrilled to be a Python core developer. I have started making patches to fix bugs in distutils, freeing Tarek to focus on distutils2. Remote teamwork was good, and now that I’ve met Alexis and other fellow Pythonistas in real life and had such good times, I know that even though the university will have to get priority this year, I will stay involved.

Thanks

First, I want to thank Google for this incredible experience we’ve lived. Their Open Source department and especially Carol Smith are doing a great job and deserve warm applause.

I am also deeply grateful to the Python Software Foundation. This summer has been amazing, and I’m very happy to be part of a project with such excellent people, to work on such an excellent programming language.

Tarek Ziadé has been a terrific over-mentor. He took the time to phone me to answer a ton of questions, he helped us keep on track with the weekly meetings, he gave trust and praise. Thank you for everything.

C. Titus Brown is modest and says that he’s not done much to mentor me. Facts don’t matter; I needed guidance more than once, and he was here to give it. Sincere thanks to you.

Many people have supported me: Shashwat Anand, without whom I would not have applied at all, roommates, friends and family. The job has not been easy each week, but it was great to be given a chance even without formal computer science or engineering background, and to get evaluations based on work. It also meant something at a personal level; I’ll talk about that in another blog post.

I’ve saved the place of honor for the other distutils2 students: Alexis, Josip, Konryd, Zubin, Mouad (not in the distutils2 group but keeping in touch weekly). I’ve found the atmosphere in our group (students, volunteers, mentors, experts) excellent, without personal conflicts or flamewars. We worked and had fun, we helped each other, we cared about each other. Sharing this experience with you all was a pleasure and an honor.

About these ads
This entry was posted in GSoC and tagged , . Bookmark the permalink.

2 Responses to Final Report: A Distutils2 Summer of Code

  1. Luca says:

    Thank you for sharing your thoughts!
    May I ask you something I can’t understand from Distutils2 docs?
    Docs says that you can specify suite or runner for your tests (in order to run them with pysetup run test) in your setup.cfg in [test] section using tags “suite” and “runner”.
    Anyway Docutils2 doesn’t seem to care too much of what I write in this section and only runs tests with “pysetup run test –suite=module_that_runs_tests”. If I don’t specify –suite or –runner in command line, Distutils2 doesn’t run any test.
    Thank you for your help and your work,
    Luca-

  2. merwok says:

    This is strange, as options are strictly the same in config files or on the command line. Care to open a bug report on bugs.python.org with more detail?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s