The implementation of the class projects is organized around a series of sprints. Each sprint is approximately one month in duration, which is longer than recommended by many agile processes but much more appropriate given the (relatively) limited time available for a class project. The sprints are designed to provide experience with agile practices, and as such have the following deliverables:

  • Specification
  • Implementation (e.g., source code)

  • Retrospective

    • Issues encountered
    • Proposed (process) changes for the next sprint

Each deliverable is described in more detail in the following sections.

Reviews and Iterative Development

Probably the best real-life software engineering lesson for me personally is the amount of scrutiny applied to proposed changes to production software. There’s excellent rationale for doing so: In many cases, the original author(s) of the code will be those who must maintain it, and consequently, the code must be not only functional but also understandable and extensible.

That perspective is very different from traditional assignments where students submit whatever they’ve completed when the due date arrives (and the maintainability of the code takes a backseat to its functionality). For this project, though, my reviews of proposed changes (in the form of pull requests) mimics my prior experiences contributing to open source projects. Unfortunately, I’m not seeing the kind of back-and-forth that I’d expect in response to my comments. In particular, I’d expect teams to address my feedback fairly quickly, and the more that you address, the higher your grade for the sprint deliverables (i.e., specification, implementation, and retrospective). This feedback is essentially a free opportunity for teams to improve their deliverables before I ever grade them as I highlighted on the prior retrospective.

In a couple of cases, the underlying issue is that it takes me a day or so to provide feedback, but there’s also cases where I’m also not seeing pull requests until just before the sprint ends. I’ve been in that situation myself for a number of projects. Nevertheless, it means that many of the proposed changes still require substantial revisions before they could be “approved” by a project maintainer. In the real world, that means that they aren’t going to be part of the next release (because they aren’t yet ready); in this class, that means that I’m awarding only 50%-ish of the total points.

Takeaway: Use the iterative development cycle to your advantage. Address feedback on pull requests as quickly as possible. To receive full credit when grading the sprint deliverables, your proposed change should be approved and merged prior to the end of the sprint.

Specification

A specification formally defines what you expect to complete in each sprint. That is, the specification is an efficient way to communicate your work with others and ensure that you have consensus on the design before implementation begins in earnest.

[…] on any non-trivial project (more than about 1 week of coding or more than one programmer), if you don’t have a spec, you will always spend more time and create lower quality code. ~ Joel Spolsky

Specifications generally cover the following information:

Overview
At a high level, what are you trying to do? Why is it important?
Problem
Describe the problem in greater detail. What are the functional and non-functional requirements of the solution? It is often very helpful to include a use case that describes the issue from a user’s perspective.
Design
What specifically do you intend to do? Here is where you include technical details: someone else who reads your specification and is familiar with the system ought to be able to implement the design without any additional information. It’s also helpful to identify alternatives for consideration, both to show that you’ve thought about them and in case your initial plan doesn’t pan out as you intend.
Work Items
Who is responsible for the work? What is the breakdown of tasks?
Testing
How do you intend to show that you’ve addressed the problem?
Documentation
What updates must be made to the system’s documentation so that others benefit from the work? Source code comments are generally expected, but user-facing functionality requires documentation in other formats for the new capabilities to be useful.

Each team is responsible for collaborating on a single specification. It is recommended that you complete your specification early in the sprint so that you have time to solicit feedback and allow time to (possibly) change its design and implementation.

The specification template is derived from those used by OpenStack. If struggling to understand what amount of detail to include in the specification, one approach is to review a a few OpenStack specifications to get a feel for what is required to manage a large open source project. Some specific examples that I oversaw while contributing to OpenStack follow:

If you skim these examples, you’ll see that the amount of detail varies depending on the complexity of the change(s). For example, the specification for ephemeral storage encryption is very high-level because of prior work on similar features whereas certificate validation is extremely detailed. In general, a more detailed specification makes the implementation easier to review and results in more rapid approval of the corresponding code changes.

A more entertaining resource is Joel Spolsky’s series “Painless Functional Specifications,” which addresses this topic humorously and pragmatically.

Implementation

Project artifacts will reside in a Git repository, which provides permanent record of all changes. The basic workflow is

  1. fork the project repository,
  2. clone your fork,
  3. make changes, and
  4. create a pull request for those changes.

If you aren’t comfortable with Git, then Pro Git is an excellent resource that not only walks through its basic use but also goes into considerable depth on its internals.

When committing anything to the repository, remember that each “logical change” should appear in its own commit. A logical change may be a complete specification, the cleanup of existing source code (e.g., reformatting it to follow an agreed-upon coding standard), or the addition of a feature. In general, the following should be avoided:

Mixing functional and non-functional changes
Create separate commits for the non-functional change (e.g., code cleanup) and the functional change; starting with the former ought to simplify the latter.
Mixing two unrelated functional changes
Break the changes apart into separate commits.
Having a giant commit for a feature
Start with the minimum functionality that provides business value (i.e., value to the end user) and add capabilities incrementally.
Mingling auto-generated with hand-written code
Often a gotcha when using web frameworks, clearly delineate the project initialization (e.g., the scaffolding provided by the framework) from its project-specific modifications to aid with review.

Typically this means that each commit has fewer than 200 lines of changes. Why is this important? Larger commits are more difficult to review, increasing the likelihood of mistakes. In addition, knowing the context for the change (e.g., who changed the line and why it was modified) is often critical to maintain the system’s functionality during later refactoring.

All changes (specifications, source code, documentation, etc.) must be reviewed prior to acceptance. Reviews ensure that others understand and agree with the proposed changes – another set of eyes often finds mistakes that the original author(s) missed!

Retrospective

During the last week of each sprint, your team should hold a retrospective to discuss the good, the bad, and the ugly. The retrospective is an opportunity for reflection. Think about the issues that your team encountered during the development cycle, consider their cause(s), and discuss possible resolutions. The retrospective is not the place for finger-pointing; rather, it is an invitation to explore how to improve the process so that the team is more successful the next iteration. (If individuals aren’t contributing equally to the project, the cause may be related to the rest of the team or the process, and soliciting their perspective is critical; otherwise, it’s appropriate to schedule a meeting with the instructor to address this issue.) Also be sure to celebrate the team’s achievements and what went well. There may even be additional opportunities for improvement in these areas.

You’ll probably benefit from appointing a scribe to take notes during the retrospective. Organize these notes into short synopsis (1/2 – 1 page) of the sprint as a whole and what would make the team more effective in the future.

A major goal of a retrospective is to enumerate issues and lessons learned. To do so effectively requires being specific. For example, it’s great to record that you started working too late in the sprint, but how late was “too late”? Maybe it was the night before or possibly the week before the end of the sprint. While that information is obvious to you now, it won’t be in a month or two. Stated differently, there should be sufficient detail to prevent another team from experiencing the same issue in the future.

Likewise, be specific when describing changes for future sprints. It’s difficult to measure improvement without having a specific objective. Using the prior example, stating that you will start working earlier in the sprint is great, but what does that mean practically? An hour earlier technically satisfies this objective but practically will make little difference. As an instructor, I want to see realistic objective setting where satisfying those objectives will improve your effectiveness as a development team. There’s also no need to sugarcoat the lessons learned: if starting the night before was a bad idea, then admit it and move forward. That admission, by itself, will not decrease your grade.

Takeaway: Be specific in the sprint retrospective.

Resources

Scott Chacon and Ben Straub, Pro Git, Apress, 2nd ed., 2014

Mike Cohn, “A Simple Way to Run a Sprint Retrospective,” Mountain Goat Software (blog), 26 January 2016

Joel Spolsky, “Painless Functional Specifications,” Joel on Software (blog), October 2000

OpenStack, “Git Commit Good Practice,” OpenStack wiki, accessed 27 September 2018

Pull Requests

A pull request is a solicitation for a project’s maintainer to accept a change from another contributor. In some cases, the pull request is accepted immediately, without any modifications; more commonly, though, the project’s maintainer reviews the proposed change and requests modification prior to accepting (and merging!) the pull request.

Best Practices

Use lots of them! In particular, logically separate changes should appear in separate pull requests. Examples of logically separate changes include commenting existing code, implementing a new capability, and adding feature specification.

Disentangling logically-separate changes from a series of commits can be challenging. I’m always available to help if you have questions about this process.

Takeaway: Create a separate pull request for every change that you want to make.

As you’re working, you’ll create pull requests, which typically have comments that should be addressed prior to being accepted. It’s unlikely to have additional changes accepted when they build on existing changes. For example, a 500-line pull request that builds on an open 200-line pull request won’t be approved until after the smaller one. Consequently, receiving approval for any open pull requests should take immediate priority.

As you think about additional capabilities to implement, focus on the minimum viable product. That is, what must be added before a typical user could take advantage of your work? Better to complete a small capability than have something that is unreliable or even unusable.

Takeaway: Try to close outstanding pull requests as quickly as possible. Aim to complete a minimum viable product by the conclusion of this sprint.

Process

“Making a Pull Request” provides more information about pull requests in Bitbucket. I recommend reviewing that tutorial for additional details regarding the process, but the general steps are as follows:

  1. A developer creates the feature in a dedicated branch in their local (clone of the) repository.
  2. The developer pushes the branch to their fork of the repository on Bitbucket.
  3. The developer opens a pull request via Bitbucket.
  4. The rest of the team reviews the code, discusses it, and modifies it.
  5. The project maintainer merges the feature into the official repository and closes the pull request.

(We are using a “Forking Workflow” so be sure to follow that description in the “How it works” section when reading the tutorial.) For another perspective on forking and pull requests, see “GitHub - Contributing to a Project” in Pro Git (Chacon and Straub, 2014), which provides a nice overview of these concepts albeit focused on using GitHub instead of Bitbucket. You can also review the configuration management lesson for more information about using Git.

The previously-described process is the one that should be used for all pull requests, including each sprint’s specification and implementation changes. Because these concepts are foundational for large-scale software projects, they’re critical to understand and be able to execute on your own. Please schedule extra instruction (EI) if you’re having trouble or would like to walk through the process with me.