Background

Silver-Platter makes it easier to publish automated changes to repositories. However, in its default mode, the only option for reviewing changes before publishing them is to run in dry-run mode. This can be quite cumbersome if you have a lot of repositories.

A new “batch” mode now makes it possible to generate a large number of changes against different repositories using a script, review and optionally alter the diffs, and then all publish them (and potentially refresh them later if conflicts appear).

Example running pyupgrade

I’m using the pyupgrade example recipe that comes with silver-platter.

 ---
 name: pyupgrade
 command: 'pyupgrade --exit-zero-even-if-changed $(find -name "test_*.py")'
 mode: propose
 merge-request:
   commit-message: Upgrade Python code to a modern version

And a list of candidate repositories to process in candidates.yaml.

 ---
 - url: https://github.com/jelmer/dulwich
 - url: https://github.com/jelmer/xandikos

With these in place, the updated repositories can be created:

 $ svp batch generate --recipe=pyupgrade.yaml --candidates=candidate.syml pyupgrade

The intermediate results

This will create a directory called pyupgrade, with a clone of each of the repositories.

$ ls pyupgrade
batch.yaml  dulwich  xandikos

$ cd pyupgrade/dulwich
$ git log
commit 931f9ffb26e9143c56f20e0b85e6ddb0a8eee2eb (HEAD -> master)
Author: Jelmer Vernooij <jelmer@jelmer.uk>
Date:   Sat Feb 25 22:28:12 2023 +0000

Run pyupgrade
diff --git a/dulwich/tests/compat/test_client.py b/dulwich/tests/compat/test_client.py
index 02ab6c0a..9b0661ed 100644
--- a/dulwich/tests/compat/test_client.py
+++ b/dulwich/tests/compat/test_client.py
@@ -628,7 +628,7 @@ class HTTPGitServer(http.server.HTTPServer):
         self.server_name = "localhost"

     def get_url(self):
-        return "http://{}:{}/".format(self.server_name, self.server_port)
+        return f"http://{self.server_name}:{self.server_port}/"


 class DulwichHttpClientTest(CompatTestCase, DulwichClientTestBase):
...

There is also a file called batch.yaml that describes the pending changes:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

name: pyupgrade
work:
- url: https://github.com/dulwich/dulwich
  name: dulwich
  description: Upgrade to modern Python statements
  commit-message: Run pyupgrade
  mode: propose
- url: https://github.com/jelmer/xandikos
  name: xandikos
  description: Upgrade to modern Python statements
  commit-message: Run pyupgrade
  mode: propose
recipe: ../pyupgrade.yaml

At this point the changes can be reviewed, and batch.yaml edited as the user sees fit - they can remove entries that don’t appear to be correct, edit the metadata for the merge requests, etc. It’s also possible to make changes to the clones.

Once you’re happy, publish the results:

$ svp batch publish pyupgrade

This will publish all the changes, using the mode and parameters specified in batch.yaml.

batch.yaml is automatically stripped of any entries in work that have fully landed, i.e. where the pull request has been merged or where the changes were pushed to the origin.

To check up on the status of your changes, run svp batch status:

$ svp batch status pyupgrade

To refresh any merge proposals that may have become out of date, simply run publish again:

svp batch publish pyupgrade

Silver Platter

Silver-platter is a new tool that aids in making automated changes across different version control repositories. It provides a common command-line interface and API that is not specific to a single version control system or hosting platform, so that it’s easy to propose changes based on a single script across a large set of repositories.

The tool will check out a repository, run a user-specified script that makes changes to the repository, and then either push those changes to the upstream repository or propose them for merging.

It’s specifically built so that it can be run in a shell loop over many different repository URLs.

 #!/bin/sh

 perl -i -pe \
 'BEGIN{undef $/;} s/Free Software
 ([# ]+)Foundation, Inc\., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA/Free Software
 \1Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301  USA/smg' *

 echo "Update FSF postal address."

 $ svp login https://github.com

 $ svp run --dry-run --diff ./fix-fsf-address.sh https://github.com/offlineimap/offlineimap
 Merge proposal created.
 Description: Update FSF postal address.

 === modified file 'offlineimap.py'
 --- upstream/offlineimap.py 2018-03-04 03:28:30 +0000
 +++ proposed/offlineimap.py 2019-04-06 21:07:25 +0000
 @@ -14,7 +14,7 @@
  #
  #    You should have received a copy of the GNU General Public License
  #    along with this program; if not, write to the Free Software
 -#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 +#    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301  USA

  import os
  import sys

 === modified file 'setup.py'
 --- upstream/setup.py       2018-05-01 01:48:26 +0000
 +++ proposed/setup.py       2019-04-06 21:07:25 +0000
 @@ -19,7 +19,7 @@
  #
  #    You should have received a copy of the GNU General Public License
  #    along with this program; if not, write to the Free Software
 -#    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA
 +#    Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301  USA

  import os
  from distutils.core import setup, Command

 $ svp run ./fix-fsf-address.sh https://github.com/offlineimap/offlineimap
 ...
 Reusing existing repository https://github.com/jelmer/offlineimap
 Merge proposal created.
 URL: https://github.com/OfflineIMAP/offlineimap/pull/609
 Description: Update FSF postal address.

1

$ debian-svp run ~/fix-fsf-address.sh offlineimap