New blog post: Providing KDE software updates from git for fun and profit in openSUSE
All checks were successful
continuous-integration/drone/push Build is passing
All checks were successful
continuous-integration/drone/push Build is passing
This commit is contained in:
parent
dbfbcc25a4
commit
5518983c09
GPG key ID:
4707F46E9EC72DEC
1 changed files with 98 additions and 1 deletions
|
|
@ -17,7 +17,7 @@ toc: true
|
|||
|
||||
If I look back at the [last post I made on ths blog](https://www.dennogumi.org/2019/07/new-server-away-from-kolab/)... let's say quite a lot of time has passed. The reason? Well, first of all, one would call it a lack of motivation[^1], and afterwards, the emergence of [a small yet quite annoying pathogen](https://www.ncbi.nlm.nih.gov/nuccore/NC_045512) which caused a bit of a stir worldwide. But today I'm not going to talk about viruses: perhaps some other time when you can avoid hear about it for breakfast, lunch, and dinner.
|
||||
|
||||
# OBS and KDE
|
||||
# KDE software from git and the OBS
|
||||
|
||||
Yes, today I'm going to talk about the [OBS](https://openbuildservice.org), that is the Open Build Service, not to be confused with [another highly successful open source project](https://obsproject.com/).
|
||||
|
||||
|
|
@ -25,6 +25,8 @@ As you know, since ages, the openSUSE KDE team provides a [series of repositorie
|
|||
|
||||
But the question that I've seen every now and then is... how it is actually done? Is everything provided by the OBS, or does someone need to add some glue on top of that?
|
||||
|
||||
# Using the OBS to provide updates to KDE packages from git
|
||||
|
||||
## Source services
|
||||
|
||||
From the [official documentation](https://openbuildservice.org/help/manuals/obs-user-guide/cha.obs.source_service.html):
|
||||
|
|
@ -83,4 +85,99 @@ Or there's a button in the OBS web interface which can allow you to do just that
|
|||
|
||||
{{< imgthumb src="images/2021/obs.png" size="600x" >}}
|
||||
|
||||
There's just a *little* problem. When there are more than 200 packages to handle, updating in this way gets a complicated. Also, while the OBS is smart enough to avoid updating a package that has not changed, it has no way to tell you before the service is actually run.
|
||||
|
||||
## Automation
|
||||
|
||||
Luckily, the OBS has features which, used with some external tools, can solve the problem in a hopefully effective way.
|
||||
|
||||
Since 2013 the OBS can use [authentication tokens](https://openbuildservice.org/2013/11/22/source-update-via_token/) to run source services, and you can for example [trigger updates with pushes to GitHub repositories](https://kamarada.github.io/en/2019/03/19/integrating-the-open-build-service-with-github/). To perform this task for the KDE:Unstable repositories, I based upon these resources to build something to do mass updates in a reliable way.
|
||||
|
||||
First of all, I generated a token, and I wanted to make sure that it could *only* trigger updates. Nothing more, nothing less. This was fairly easy with `osc`:
|
||||
|
||||
```
|
||||
osc token --create -o runservice
|
||||
```
|
||||
|
||||
I did not specify a project, so the token works with all the repositories I have access to. This was a requirement, because the KDE Unstable repositories are actually three different OBS projects.
|
||||
|
||||
To trigger an update in the OBS, one needs to call the `https://api.opensuse.org/trigger/runservice` endpoint, doing a POST request and include the project name (`project` parameter) and the package name (`package` parameter). An example (I know, I didn't encode the URL for simplicity, bear with me):
|
||||
|
||||
```
|
||||
https://api.opensuse.org/trigger/runservice?project=KDE:Unstable:Frameworks&package=kcoreaddons
|
||||
```
|
||||
|
||||
The token needs to be passed as an `Authorization` header, in the form `Token <your token here>`. You get a 200 response if the trigger is successful, and 404 in other cases (including an incorrect token).
|
||||
|
||||
Like this, I was able to find a way to reliably trigger the updates. In fact, it would be probably easy to conjure a script to do that. But I wanted to do something more.
|
||||
|
||||
## A step further
|
||||
|
||||
I wanted to actually make sure to trigger the update *only* if there were any new commits. But at the same time I did not want to have a full checkout of the KDE git repositories: that would have been a massive waste of space.
|
||||
|
||||
Enter `git ls-remote`, which can give you the latest revision of a repository for all its branches (and tags), *without* having to clone it. For example:
|
||||
|
||||
```
|
||||
git ls-remote --heads https://invent.kde.org/pim/kitinerary.git/
|
||||
22175dc433dad1b1411224d80d77f0f655219122 refs/heads/Applications/18.08
|
||||
5a0a80e42eee138bda5855606cbdd998fffce6c0 refs/heads/Applications/18.12
|
||||
2ca039e6d4a35f3ab00af9518f489e00ffb09638 refs/heads/Applications/19.04
|
||||
2f96d829f28e85f3abe486f601007faa2cb8cde5 refs/heads/Applications/19.08
|
||||
f12f2cb73e3229a9a9dafb347a2f5fe9bd6c7975 refs/heads/master
|
||||
18f675d888dd467ebaeaafc3f7d26b639a97d142 refs/heads/release/19.12
|
||||
90ba79572e428dd150183ba1eea23d3f95040469 refs/heads/release/20.04
|
||||
763832e4f1ae1a3162670a93973e58259362a7ba refs/heads/release/20.08
|
||||
c16930a0b70f5735899826a66ad6746ffb665bce refs/heads/release/20.12
|
||||
```
|
||||
|
||||
In this case I limited the list to branches to branches (`--heads`). As you can see, the latest revision in `master` for kitinerary at the time of writing is `f12f2cb73e3229a9a9dafb347a2f5fe9bd6c7975`. So, the idea I had in mind was:
|
||||
|
||||
1. Get the state of the master branch in all repositories part of the KDE Unstable hierarchy;
|
||||
2. Save the latest revision on disk;
|
||||
3. On the following check (24 hours later) compare the revisions between what's stored on disk and the remote;
|
||||
4. If the revisions differ, trigger an update;
|
||||
5. Save the new revision to disk;
|
||||
|
||||
This was slightly complicated by the fact that package names on the OBS and source repositories in KDE can differ (example: `plasma-workspace` is `plasma5-workspace` or `akonadi` is `akonadi-server`). My over-engineered idea was to create a JSON mapping of the whole thing (excerpt):
|
||||
|
||||
```json
|
||||
{
|
||||
"KDE:Unstable:Frameworks": [
|
||||
{
|
||||
"kde": "frameworks/attica",
|
||||
"obs": "attica-qt5",
|
||||
"branch": "master"
|
||||
},
|
||||
{
|
||||
"kde": "frameworks/kdav",
|
||||
"obs": "kdav",
|
||||
"branch": "master"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
(Full details on the [actual repository](https://git.dennogumi.org/einar/scripts/src/commit/30b7fda620f501aea68a02968784167e91636e7d/obs/config/repo_config.json))
|
||||
|
||||
It was painful to set up the first time, but it paid off afterwards. I just needed a few more adjustments:
|
||||
|
||||
- Check whether the remote repository actually existed: I used [GitLab's project API](https://docs.gitlab.com/ee/api/projects.html) to obtain a yes/no answer for each repository, and skip them if they do not exist.
|
||||
- Commit the changed files to git and push them to the remote repository holding the data.
|
||||
|
||||
As I am mostly a Python person, I just used Python to write a [yet another over-engineered script](https://git.dennogumi.org/einar/scripts/src/branch/master/obs/update_unstable.py) to do all the steps outlined above.
|
||||
|
||||
## Icing on the cake
|
||||
|
||||
Mmm... cake. Wait. We're not talking about food here, just about how the whole idea was put into "production" (include several hundred of quotes around that word). I created a separate user on my server (the same which runs dennogumi.org) with minimal privileges, put the token into a file with 0600 permissions, and set up a cron job which runs the script every day at 20 UTC+1.
|
||||
|
||||
There was just one extra step. Historically (but this is not the case anymore nowadays) the OBS would fail to perform a git checkout. This would leave a package in the **broken** state, and the only way to recover would be to force an update again (if that did not fix it, it would be time to poke the friendly people in `#opensuse-buildservice`).
|
||||
|
||||
Inspired by what a former KDE team member (Raymond "tittiatcoke" Wooninck) did, I wrote [a stupid script](https://git.dennogumi.org/einar/scripts/src/branch/master/obs/fix-broken.sh) which checks the packages in broken state (at the moment just for one repository and one architecture: a broken clone would affect all of them equally) and forces an update. It needs to use tokens rather than `osc`, but I'll get to that soon, hopefully.
|
||||
|
||||
# Conclusion
|
||||
|
||||
There, you have it. This is how (at the moment) I can handle updating all KDE packages from git in the OBS with (now) minimal effort. Probably it is an imperfect solution, but it does the job well. Shout out to Fabian who mentioned tokens and prompted the idea.
|
||||
|
||||
|
||||
[^1]: Also called laziness.
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue