Get know how to solve issue with pushing to submodules directly from the main repo while keeping the project easily cloneable by external contributors.
The Git submodules mechanism is pretty handy to keep the source code of lousily related dependent software together in one Git repository while leaving their development separate. It is something like symlinks in the Unix world, but with an ability to refer also to the previous version. It’s quite popular in the projects using source code integration (instead of shared libraries) or to speed up development by making related changes in multiple repositories easier. It is not the only possible solution – Gradle, for instance, provides a composite build mechanism. Monorepo is an another approach, but it has its own limitations and it very problematic to use in FOSS projects developed by different people/teams.
As usual, I encountered that situation in one of my projects. As a big enthusiast of automatic code testing and Continuous Delivery, some time ago I have been working on improving the reliability of my (automatically released) gradle-nexus-staging-plugin. After each commit (so also before the release) I wanted to have the end-to-end tests executed to verify that the plugin is able to pass a simple (but real) project through the release process to Maven Central (aka The Central Repository).
I could move that test project to the plugin repository, but – well – it’s a distinct project which could be also released separately or replaced with some other project. In addition, testing two variants of releasing it was handy to use keep it in two branches “mounted” twice in my root repository.
Problem (aka Challenge)
gradle-nexus-staging-plugin contains a submodule
nexus-at-minimal from a separate repo. Working on new fancy feature in GNSP it is needed to tweak the acceptance testing project. To make the development smoother it’s very useful to be able to commit introduces changes in the dependent project directly from the working copy of the main project. It works out of the box. However, later on we should also directly push them back to a separate repo of that dependent project (by stepping into the submodule and calling
git push or – all-in-once by – using the
git push --recurse-submodules=on-deamand parameter when pushing from the main repository). And then – in the long term – we encounter some inconvenience.
Let’s start by defining a submodule path to connect via SSH:
In general it works fine and pushing is allowed. However, any non-developer trying to clone that repo (and initialize submodules) gets:
It’s pretty bad in the open source (FOSS) development where the projects are publicly available other people are encouraged to download (clone) it, build and contribute.
The obvious remedy to the previous error is switching to HTTPS:
However then, to allow developers to commit changes directly to a submodule it is required to go through a separate HTTPS authentication which in the most cases is completely not use in favor of SSH (and would need to be configured independently with an access token).
To keep external contributors happy the developers could manually change the url in
.gitmodules from HTTPS to SSH for every cloned project. However, it’s quite tedious. The better solution is an usage of
pushInsteadOf. For the aforementioned example, the developers need only to add to the global
~/.gitconfig configuration file:
It effectively overrides the push URL to use SSH instead of HTTPS for the whole group in GitLab/GitHub, covering also submodules (where HTTPS – external contributors friendly scheme – is left as a default).
URL rewriting and conditional configuration (covered in the first part) are just a subset of options available in Git to make the development more flexible and simple. Simple, assuming we already found the required features and learned how to use it ;-).
Published on Java Code Geeks with permission by Marcin Zajaczkowski, partner at our JCG program. See the original article here: Easier work with Git submodules with in-place push access – Git Tricks #2
Opinions expressed by Java Code Geeks contributors are their own.