Country Industrial

My work at Canonical includes a lot of time spent on Landscape, mostly from the end-user support perspective. It’s billed as a single pane of glass for managing Ubuntu hosts and enabling change control at scale, including on-premises repository mirroring and filter-based pull pockets for staging and rolling out upgrades across an estate. The official documentation surveys mirroring the Ubuntu repos fairly well, and the default tooling expects repository structures to match the Ubuntu ones, but information is a little more scant when it comes to mirroring third-party repos. This is something I do with my on-premises deployment, both for distributing third-party software and for mirroring entire other distributions than Ubuntu, like the official Debian Bookworm mirrors for my Raspberry Pis.

Getting started

I am going to assume that you already have a working deployment of Landscape, version 23.03 or greater. The official deployment guides cover this – see the quickstart, Juju, or manual instructions – so there is no need to rehash that here. You should also read through the repo mirroring docs and at least have your signing key imported, or preferably have an Ubuntu repo mirrored to get a sense for the process. If you deployed on Jammy/Noble or have the app server on a container running those, you will want to check out the RabbitMQ tuning guide as well to avoid having your pocket syncs fail.

There are a few more things I maintain in my lab that I recommend, but they’re not requirements:

• Certificate installed on the Landscape server signed by your internal CA
• Some kind of DNS infrastructure; it’s much easier resolving hostnames to IPs when dealing with all this
• Prometheus or similar agents reporting on any hosts you plan to register

Debian

Because the archives are so similar, mirroring the upstream Debian repo is fairly straightforward; there are just a few differences between it and the Ubuntu archives. Landscape automatically installs and configures your mirrors with the Ubuntu archive signing keys, so you don’t need to pass one to the API when creating your distro, series, and pockets; the same isn’t true of third party repos. Debian also splits its security repo off into its own suite signed by a different key, so you’ll need to add that pocket to the series manually.

Importing the keys

All of the Debian signing keys are available from the ftp-master page. Since my Pis run Bookworm, we’ll use the Bookworm keys demonstratively.

Like always, please check the fingerprint of the key before you trust it blindly! You can pass a key without options to gpg and it will print a summary:

c0w80yd4n@mazu:~$ gpg element-key.asc
gpg: WARNING: no command supplied.  Trying to guess what you mean ...
pub   rsa4096 2019-04-15 [SC] [expires: 2033-03-13]
      12D4CD600C2240A9F4A82071D7B0B66941D01538
uid           riot.im packages <packages@riot.im>
sub   rsa3072 2019-04-15 [S] [expires: 2025-03-15]

The archive signing keys are already encoded, so all you need to do is grab them from the server and then import them with the API.

curl https://ftp-master.debian.org/keys/archive-key-12.asc > bookworm-archive-key.asc

landscape-api import-gpg-key \
	bookworm-archive-key \ # Name of the key in Landscape 
	bookworm-archive-key.asc \ # File name 
	--method=POST \
	--json

I’m passing the --method=POST option here to avoid errors from the URL getting too long. The security key is next:

curl https://ftp-master.debian.org/keys/archive-key-12-security.asc > bookworm-security-archive-key.asc
landscape-api import-gpg-key \
	bookworm-security-archive-key \ 
	bookworm-security-archive-key.asc \
	--method=POST \
	--json

Because these are public keys, the return value you get back will indicate they don’t have a secret:

{
  "id": 57,
  "name": "example-key",
  "key_id": "391CB00B51011EEE7",
  "fingerprint": "this:aint:just:some:four:char:word:list:baby:girl",
  "has_secret": false
}

You’ll use these keys when you create the repository structure.

Creating the distribution, series, and pockets

Creating a distribution is incredibly straightforward, as they are quite literally just names.

landscape-api create-distribution debian

Creating a series, however, can trip people up. It helps to remember that you’re recreating what you would see in a sources.list entry:

deb MIRROR_URI/DISTRIBUTION SERIES COMPONENTS
deb http://deb.debian.org/debian bookworm main non-free-firmware non-free
deb MIRROR_URI/DISTRIBUTION SERIES-POCKET COMPONENTS
deb http://deb.debian.org/debian bookworm-updates main non-free-firmware non-free

A note on mirrors

I’m just using the deb.debian.org/debian mirror URI for demonstrative purposes. It may prove worthwhile to explore the mirror options in your area and sync from those URIs directly; this also helps mitigate the risk posed by unsynchronized mirrors to your mirror if your sync operation resolves to a different place in the middle of the job.

Back to business

So, remembering that the release pocket is implied, the API call to create our Bookworm series along with the release and updates pockets will look like:

landscape-api create-series bookworm debian \
	--mirror-uri http://deb.debian.org/debian \
	--components main,non-free-firmware \
	--architectures arm64 \	# Raspis; you probably want amd64.
	--pockets release,updates \ # Security comes later
	--mirror-series bookworm \
	--mirror-gpg-key bookworm-archive-key \	# Upstream key
	--gpg-key your-mirror-key # Your key; both are necessary

Because the security pocket is signed with another key, and its upstream URI differs from the main archive, let’s create it separately here.¹

The syntax for creating a pocket is a bit different, with the Debian source entry bits being passed as positional arguments, with an additional argument for the pocket’s mode. Options passed are for Landscape to grok the upstream mirror’s repo structure and use the correct key:

landscape-api create-pocket \
	security \ # Name
	bookworm \ # Series
	debian \ # Distro
	main,non-free-firmware \ # Components
	arm64 \ # Arches
	mirror \ # Mode
	your-mirror-key \ # Mirror signing key
	--mirror-uri http://security.debian.org/debian-security \
	--mirror-gpg-key bookworm-security-archive-key \
	--mirror-suite bookworm-security

We now have:

• A debian distribution
• With a bookworm series
• Containing the pockets release and updates, signed by bookworm-archive-key
• And the pocket security signed by bookworm-security-archive-key
• With each pocket mirroring the components main and non-free-firmware

With the repository structure in place, you can now synchronize each pocket with the upstream mirror.

Synchronizing the pockets

Pocket syncs look no different from their Ubuntu counterparts:

landscape-api sync-mirror-pocket release bookworm debian
landscape-api sync-mirror-pocket updates bookworm debian
landscape-api sync-mirror-pocket security bookworm debian

Synchronize each pocket one at a time, and look on in awe as absolute mountains of disk space are consumed automagically. Isn’t technology great?

Registering Debian clients

Generally speaking, you should be able to register Debian clients with Landscape with a plain-Jane install of the client from the Landscape beta PPA. You’ll need to import those repo keys as well, but this is handled elegantly by gpg for you – then just add the sources line, update, install, and run:

gpg --keyserver keyserver.ubuntu.com --recv-keys 6e85a86e4652b4e6

gpg --export 6e85a86e4652b4e6 | sudo tee -a /usr/share/keyrings/landscape-client-keyring.gpg > /dev/null

echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/landscape-client-keyring.gpg] https://ppa.launchpadcontent.net/landscape/self-hosted-beta/ubuntu focal main" | sudo tee -a /etc/apt/sources.list.d/landscape-client.list

sudo apt update
sudo apt install landscape-client
sudo landscape-config

This is supposed to work fine, but I encountered two separate problems on my Pi that I don’t want to admit are probably skill issues, so I’ll cover them here. On my Plex server, the configuration step failed on account of a missing Python dist-package dependency:

File "/usr/bin/landscape-config", line 12, in <module>
   from landscape.client.configuration import main
 File "/usr/lib/python3/dist-packages/landscape/client/configuration.py", line 20, in <module>
   from landscape.client.broker.config import BrokerConfiguration
 File "/usr/lib/python3/dist-packages/landscape/client/broker/config.py", line 4, in <module>
   from landscape.client.deployment import Configuration
 File "/usr/lib/python3/dist-packages/landscape/client/deployment.py", line 14, in <module>
   from landscape.client.snap_utils import get_snap_info
 File "/usr/lib/python3/dist-packages/landscape/client/snap_utils.py", line 3, in <module>
   import yaml
 No module named 'yaml'

This was trivial to resolve:

sudo apt install python3-yaml

It turns out that Ubuntu and Debian differ in their priority marking for the python3-yaml package, so one ships with the base install while the other does not:

~$ lsb_release -d 2>/dev/null
Description:    Ubuntu 23.10
~$ apt-cache show python3-yaml | grep Priority
Priority: important

~$ lsb_release -d 2>/dev/null
Description:    Debian GNU/Linux 12 (bookworm)
~$ apt-cache show python3-yaml | grep Priority
Priority: optional

It’s also not marked as a dependency for the Landscape client package, which sort of makes sense considering it’s asssumed to be present:

~$ apt-cache show landscape-client | grep Depends
Depends: python3:any, debconf (>= 0.5) | debconf-2.0, libc6 (>= 2.17), landscape-common (= 24.02+git6339-0ubuntu0), python3-pycurl, python3-dbus

On my Pi-hole, the problem was a little more interesting. The installation and configuration seemed to go fine, but when I went to its Packages page on the GUI, I had an OOPS thrown; instinctually I checked /var/log/landscape/monitor.log (formatted prettier for your viewing pleasure):

2024-04-14 00:51:12,790 WARNING  [MainThread] Package reporter output:
Traceback (most recent call last):
  File "/usr/bin/landscape-package-reporter", line 12, in <module>
    main(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/landscape/client/package/reporter.py", line 945, in main
    locale.setlocale(locale.LC_CTYPE, ("C", "UTF-8"))
  File "/usr/lib/python3.11/locale.py", line 626, in setlocale
    return _setlocale(category, locale)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
locale.Error: unsupported locale setting

Locale issues; classic! The first thing to do is check what locale is actually reporting; it’s used by a lot of services, so you can usually peek in the journal:

/etc/default/locale: No such file or directory

Okay… can I just cheat, and export the environment variable manually?

export LC_ALL=C

The answer is yes, albeit temporarily. The mechanism behind UTF-8 encoding and its interaction with locale is way out of scope for this adventure, but might make a good topic for later. If you encounter this issue, it’s because the Landscape client is trying to ensure the package reporter doesn’t return a stream of Hot Garbage™ instead of intelligible text as a consequence of mangling and other shenanigans that locale settings influence when it comes to output:

def main(args):
    # Force UTF-8 encoding only for the reporter, thus allowing libapt-pkg to
    # return unmangled descriptions.
    locale.setlocale(locale.LC_CTYPE, ("C", "UTF-8"))

libapt-pkg uses C string manipulation functions, and locale settings make its outputs non-deterministic, whereas a strict Unicode reading of values avoid the possibility that an uppercase “I” doesn’t get turned into the relatively-uncommon-in-packaging character “ı” because you’re in Turkey.

This is another instance where I needed to install a package:

sudo apt install locales

Then reconfigure, making sure to tick the box en_US.UTF-8, and then mark the C UTF-8 locale as default for the system:

sudo dpkg-reconfigure locales

For the adventurous, the Landscape client snap just made its way into general release. I won’t go into the details of installing snapd on Debian which can be its own ordeal, but any feedback from those willing to give it a spin is welcome.

Associating repository profiles

To actually get your Debian clients to use the mirrors you put your blood, sweat, and tears into creating, you’ll need to associate them with a repository profile. Repo profiles are a rather slick way to handle apt configuration across your hosts – when you create them, try to give them as meaningful of a name as possible, since they’re easy to switch out and the flexibility they afford is too valuable to waste:

landscape-api create-repository-profile bookworm-raspi \
	--access-group global \ # This is another useful tunable
	--description "Bookworm main repo mirror for service raspis"

Add the pockets we created earlier to the profile:

landscape-api add-pockets-to-repository-profile \
	bookworm-raspi \ # Profile name
	release,updates,security \ # Pockets
	bookworm \ # Series
	debian # Distro

The repo profile association mechanism uses the tags you add to your registered clients, or they can be applied to all of your computers; how you want to structure those tags is up to you, but there’s no limit to how granular you can make them:

landscape-api associate-repository-profile \
	bookworm-raspi \ 
	--tags raspi

This creates an activity that will get delivered to the tagged hosts which will modify their sources.list to point to your mirrors. You can check the status of this with get-activities and confirm it’s behaving as expected on the hosts themselves:

$ cat /etc/apt/sources.list
# Landscape manages repositories for this computer
# Original content of sources.list can be found in sources.list.save
$ cat /etc/apt/sources.list.d/landscape-bookworm-raspi.list
deb http://gamer.zone/repository/standalone/debian bookworm-security main non-free-firmware
deb http://gamer.zone/repository/standalone/debian bookworm main non-free-firmware
deb http://gamer.zone/repository/standalone/debian bookworm-updates main non-free-firmware

Configuring additional sources

It’s up to you how you want to configure additional sources for the other components in a series, but Landscape does include an option to add arbitrary sources to a repository profile which can cover things like the non-free component I chose not to mirror (because non-free software is icky, but raspi firmware is too inconvenient to be principled about):

landscape-api create-apt-source \
	bookworm-non-free \ # Name 
	"deb http://deb.debian.org/debian bookworm non-free" \ # Apt source line
	--gpg-key bookworm-archive-key # Sign with the upstream key

Do the same for the updates pocket:

landscape-api create-apt-source \
	bookworm-updates-non-free \
	"deb http://deb.debian.org/debian bookworm-updates non-free" \
	--gpg-key bookworm-archive-key

Then add the sources to the appropriate repository profile:

landscape-api add-apt-sources-to-repository-profile \
	bookworm-raspi \
	bookworm-non-free,bookworm-updates-non-free

It’s worth noting that this mechanism can be used for arbitrary sources, so you can use it to add sources for things you don’t necessarily want mirrored, but want available on the hosts you manage. The classic example is something like VS Code; assuming the Microsoft signing key was imported:

landscape-api create-apt-source \
	visual-studio-code \
	"deb https://packages.microsoft.com/repos/code stable main" \
	--gpg-key microsoft-packages-key

landscape-api add-apt-sources-to-repository-profile \
	soydevs-cant-vim-workstations \
	visual-studio-code

I hope this helps some brave soul out there. Landscape’s mirroring and repo profiling functionality are its most powerful features – being able to skip the complexity of dealing with reprepro or the unmaintained apt-mirror is a godsend, and being able to apply it to other distros makes it even better.