In our sample game we look at playing as Austria and aiming for a Diplomatic Victory. And our focus is on puppeting City-States, but I misunderstood and instead of making a Diplomatic Victory easier, it makes it harder. I still managed to get my Diplomatic Victory, but a Science or Domination Victory would definitely have been easier in this scenario.

Playing Civilization V, Part 14

Now I am at turn 340, I just puppeted another City-State, and I have one more on the way. I have sent a large number of military units, both land and sea, to the City-States that are on Japan’s border. If he tries anything now he will likely regret it. Now I am starting to build things I had not bothered with before, like Granaries and Amphitheaters. I got a couple of Great Engineers, and used them to rush build Wonders. I am nearly finished puppeting all of the City-States, and my Happiness and Income are both quite healthy. I am in such good shape I started to build the World’s Fair, which I initally proposed then ignored.

At this point I only have one more City-State to puppet, then I will have puppeted every City-State in the game. So it is time to put my large income to use in other ways by giving gifts to the other players. I may want to get a few more votes on my side to get over the top in the World Leader vote when it happens. On turn 357 I completed the puppeting of all of the City-States in the game. Meanwhile, I am building up my Science and building Wonders as they become available. I didn’t build any in the early game because I had other priorities, but now I have a strong military, high income, and no other pressing needs. My score on the Leader Board is now double that of my nearest competitor which is still Japan. And their denunciation me of just expired with them taking no action.

I used a Great Engineer to rush build the Porcelain Tower, earning me another Great Scientist, which I then used to build another Academy. At this point it is mostly a slog to the finish. Since I was so dominant and so wealthy I could afford to pursue a couple of other options just in case. One was a Science victory, which looked like a possibility, though it would be close. I had to be very focused in my Science Research. The other option was a Culture victory, and I had dominated two other empires culturally without half trying, and I was generating Great Artists, Great Musicians, and Great Writers pretty rapidly. So I built Museums, then Archeologists, and grabbed every artifact I find to stuff into the Museums. But the final Victory came when I researched Globalization. This tech allows you to get an extra vote in a World Council for every Spy you send to another empire’s capital and designate as an Ambassador. I immediately sent my spies out (they had previously been in my cities doing counter-espionage since I had a tech lead). And a few turns later a vote for World Leader came up and I won.

This was my first time using Maria Theresa of Austria as my leader, and it was fun, but in the end puppeting all of the City-Staters actually made it harder to win a Diplomatic victory. In other games I have played, every City-State that was an ally was an extra vote in the World Council, and I assumed that making them puppets would mean I would get that many votes. But that didn’t happen. Of course, if I had read more carefully in the Civilization Wiki I would have realized this, because it says there very clearly that Austria is one of the worst Empires to pick if you are going for a Diplomatic victory. So the fact that I managed it anyway is nice. I think I actually could have done a Science victory as I had my Spaceship half-built when I won the Diplomatic Victory. But I hope this analysis of how I actually played this game will be helpful to someone else.

Links

• https://www.palain.com/gaming/civilization-v/playing-civilization-v-part-14/

Provide feedback on this episode.

Hello, again. This is Trey.

Welcome to part 8 in my Cheap Yellow Display (CYD) Project series.

If you wish to catch up on earlier episodes, you can find them on my HPR profile page https://www.hackerpublicradio.org/correspondents/0394.html

It is hard to believe that I started this project and the HPR series to document it more than a year ago. Time flies. Life happens. I spent the last 8 months so focused on work related activities that I had to set the project aside. And once I set it aside, it was difficult to get back to again. The one time I tried, I found that my son's old Windows laptop, which I had commandeered to use for the project, was once and truly dead.

We live in a different world now than we did when I began this project. Today, everything is about AI – how it is changing our world, increasing efficiencies, and even displacing certain types of jobs. "Vibe coding" is transforming the way we make software, and now everyone is a developer.

Within my organization, we are all being strongly encouraged to learn more about AI and apply it in our daily work. We are blessed to have access to a wide range of training and to powerful tools which support the process. Several colleagues within my organization and outside my organization have recommended Claude Code -- for development, for organization, for brainstorming, and for much more. My role is not that of a developer, and I have had no need for Claude Code at work. There are plenty of other tools for me to use.

But at home, I thought... I could install Claude Code at home to experiment with and to learn. And then it hit me. I wonder if I could use Claude Code to help me with my stalled CYD project.

"Hello, my name is Trey, and I am a fraud."

OK. I don't think I am a fraud, but having never used such a powerful tool to help me code, I feel a little bit like a fraud, with Claude doing the work for me. Let's talk through what we did.

As I mentioned, I was unable to use the laptop on which I created the original GUI code. But no worries, because It was all on GitHub, right?

So, I began by trying to install Claude Code on one of my Ubuntu machines. That failed miserably, and all the instructions I found online recommended running it on Windows. So I transitioned to installing Claude Code on a freshly built Windows 10 system using some old hardware which will not support Windows 11. Surprisingly, It seems that Claude wants to run within Ubuntu on Windows Subsystem for Linux (WSL).

Once I went through all the hoops for installing and setting up my basic subscription, I got to work. I created a new directory to use for all my project files, and another one for Claude to write all its output files to.

Everything I have read and heard about using Claude Code for a project like this recommended beginning by using Claude to create the project plan. So my first prompt was as follows:

"I am building a project using an ESP32 cheap yellow display. The final goal is to have a device with a graphical user interface which will allow me store several text messages, each assigned to their own specific button. Then, when one of the buttons is pressed, the message will be converted into morse code, and the message will be sent by blinking an onboard LED. I already have the initial design for the GUI coded in my github repository. Please construct a detailed step by step plan for building this application"

I wanted to keep it simple at the beginning, so even though I plan to use an opto-isolated relay for the output, I am starting out using the built in LED.

Claude asked me for the URL for my Github repository, and I provided it. After thinking for a bit, it produced a project plan.

CYD Morse Sender — Step-by-Step Build Plan

What You Already Have

- 3 fully designed LVGL screens (Send, Config, Keyboard)

- 7 send buttons with green styling on the Send Screen

- Speed slider (5–30 WPM) and program slot dropdown on Config Screen

- Text area + LVGL keyboard widget on Keyboard Screen

- Display driver and touchscreen hardware initialized (but touch read was stubbed out)

---

Phase 1: Fix Touchscreen Input

Initialize the XPT2046 on VSPI, replace the hardcoded touched = false with real touch data, calibrate raw ADC coordinates to screen pixels, and verify all three nav buttons transition screens correctly.

Phase 2: Persistent Message Storage

Use Preferences.h / NVS to store 7 messages ("msg0" – "msg6") that survive power cycles. Load on boot and populate send button labels.

Phase 3: Config Screen — Slot Selection & Speed

Wire the Program dropdown to g_selectedSlot, wire the Speed slider to g_wpm with a live WPM label, and wire the Set button to clear the text area and navigate to the Keyboard Screen.

Phase 4: Keyboard Screen — Capture and Save Text

On keyboard OK (LV_EVENT_READY): read the text area, save to NVS, update the send button label, navigate back. On Cancel: navigate back without saving.

Phase 5: Morse Code Engine

Build morse.h with the full A–Z / 0–9 lookup table, a textToMorse() converter, and Paris-standard timing (unit_ms = 1200 / wpm).

Phase 6: Non-Blocking LED Transmission

State machine (IDLE / LED_ON / LED_OFF) driven by millis() in a morseUpdate() function called every loop — no delay(). LED on GPIO 17 (blue).

Phase 7: Wire the Send Buttons

Each button calls startMorse(getMessage(slot)) if not already transmitting. Disable all send buttons during TX; re-enable when the state machine reaches the end.

Phase 8: Integration & Polish

Stop button, WPM persistence, empty-slot guard, serial debug output, final touch calibration, end-to-end test checklist.

---

Recommended order: Phase 1 → 2 → 7 (partial) → 5 → 6 → 7 (full) → 3 → 4 → 8

This was wild! Claude determined exactly how far along I was in the project, and also alerted me to the fact that the "touch read was stubbed out". Ugh. This means that the files I had uploaded to GitHub were my original GUI files (Episode 05 – HPR4532 - https://hackerpublicradio.org/eps/hpr4532/index.html ) and not the ones that I finally got working properly (Episode 07 – HPR4624 - https://hackerpublicradio.org/eps/hpr4624/index.html ). That was my own fault. Did I mention that I don't get Git? I REALLY need to learn to properly use Git!

But, we have a plan, broken down by eight numbered phases. And they seem to address all the functionality I wanted with a few additional things I had not thought about. Interestingly, even though these phases are sequentially numbered, Claud recommended that we approach them in a bizarre order: Phase 1 → 2 → 7 (partial) → 5 → 6 → 7 (full) → 3 → 4 → 8 .

Alright. Let's see what we can do. The first phase is to fix the touchscreen input.

Claude took me through it step-by-step, asking as it needed to read specific project files.

Finally, it wrote a new ui.ino code file to my speficied output directory for me to test. I copied it into the correct file location, said a quick prayer, compiled in Arduino IDE, and downloaded to the CYD.

Well, that is... interesting. The display looked nothing like it was supposed to. There were vertical green bars with smaller dashed green vertical stripes in them. I will include a picture in the show notes so that you can see what it looked like and why it was so difficult to describe.

I spent the next hour or so trying to explain what I was seeing to a chat bot. Claude recommended potential fixes which either did nothing or made the situation worse. I began questioning whether this was a good idea, how people actually gained efficiencies talking to a bot, and even several life choices.

Then I had a thought. I prompted Claude:

If I were to take a picture of the screen on the cheap yellow display and copy it into the output folder, would you be able to analyze it to better determine what is wrong and how to fix it?

Shockingly, Claude answered in the affirmative, and told me to copy the picture to the output folder and let it know when to proceed. It analyzed the picture and more of the supporting files it had copied from my GitHub, asking each time if it could access that file. It determined that my original code was written for a flavor of LVGL version 8 and I was now using LVGL 9.5.

It recommended changes, and then asked permission to make those changes, file by file. .h files & .c files, Finally, I just gave it permission to edit the files in the project folder without asking for permission for each file each time. Claude was still explaining each change, showing me exactly what would be changed, and asking for permission, so that I could review all of the changes. But now it was not asking additional permission to write to each of the impacted files.

Next, Code compiled and downloaded. Different screen, but not right. Again, I took a picture and gave it to Claude to analyze. So, Claude paused and altered the code to generate a specific test pattern overtop of the GUI.

The test pattern was supposed to cover the entire rectangular screen. But parts of the pattern were in a square on the screen and parts were not. Another photograph and analysis, told Claude that there were some rotation/screensize issues.

We repeated this several times. Some resulted in improvement, and others did not.

This is the point where I noticed something interesting. Not about Claude, specifically, or about the app. But I noticed something interesting about myself and about the process.

Previously, when I was working through some of these challenges without Claud, I found myself becoming more and more stressed, frustrated, and angry, until I found a solution. Then another problem would repeat the cycle. Success in the end was great, but the emotional extremes during the process were not always pleasant.

Now, I was effectively managing the project, and relaying information to the resource responsible for fixing the problems -- a very different experience.

But I also ran into another issue. Claude became absolutely certain that the problem revolved around the device not accurately knowing where the 4 corners of the screen were. But in reality, the output of the test pattern was rotated 90 degrees from the actual screen. It took several iterations of me insisting that the problem had to do with screen orientation and not corner coordinates. It was interesting to experience the tool doubling down on an obvious mistake, but we finally resolved that.

Again, while it was frustrating, it was much less stressful.

We proceeded to Phase 2: Persistent Message Storage where we ensured that the button labels on the send screen were stored in the devices persistent storage, so that, when they are edited to contain the message they should send, that information would survive a reboot.

Next, we combined elements of Phase 5: Morse Code Engine , Phase 6: Non-Blocking LED Transmission , and Phase 7: Wire the Send Buttons together. Building the morse code engine was an area I had been thinking about for a while. I already had working parts of something similar in the Arduino practice oscillator I have referenced a few times in this series. The code for the practice oscillator may be found on my GitHub, but it was all based on original code from jmharvey1, with my only contribution being making pin assignments variables so that the code could easily be ported to different devices.

So, I was happy that we were building the morse code engine directly. The code for it may be found in morse.h, which uses a constant character lookup table to define each character. Without any specific direction from me, Claude used the PARIS timing methods I have already described within Episode 6 of this series. It defines timing for DOT, DASH, LETTER_GAP, and WORD_GAP, and all are based on a simple calculation of 1200 ms / the number of words per minute (WPM) we wish to transmit.

Along the way, we discovered that, if we tried to use the delay() function, it would crash the program due to a conflict with the LVGL timer used for touchscreen inputs. Claude altered all the delays accordingly.

Then, Phase 3: Config Screen — Slot Selection & Speed allowed us to configure the WPM we wished to use in addition to selecting a specific Send button to reconfigure. This forced us to work on Phase 4: Keyboard Screen — Capture and Save Text which is used to type the entries for each Send button. At this point, I also decided that we would want to also use the Keyboard Screen to send ad hoc morse as we typed it.

During this phase we discovered several bugs which seemed to cause random freezes. Careful troubleshooting with messages output to the Arduino IDE's serial console helped us narrow down the causes and remedy them.

Finally all the tests worked and I am able to merrily pre-configure macro buttons with custom messages and use the CYD to send the morse code for those messages to the on-board LED at whichever rate I specify.

I have noticed in my presentation of this narrative that I repeatedly slip into the first person plural terms "we" and "us" instead of the first person singular terms "I" and "me". I have unconsciously personified Claud and recognized it as an integral part of my (formerly one person) development team.

I finally configured Claude to connect to my GitHub repo and upload all the files and documentation. We additionally created a CYD-Narrative.md file which describes in more detail all the work which was done on the project. I still do not 100% get git, but we are successfully using it.

You can find all these files in my GitHub repo ( https://github.com/jttrey3/CYD_MorseSender ) where they are shared under a GPL 3.0 license.

There are still several additional steps I plan to complete in the next few months.

1. I will be integrating an opto-isolated relay which will allow me to plug the device into the straight key input on any amateur radio. This will require a battery power source, charge controller, and more hardware.

1. I... make that "We" (Claude & I) will be modifying the code to support an audio side tone through an attached speaker when sending code
2. We will add an output selection switch to the config page to choose any combination of speaker, relay, or LED as output.
3. We will develop a downloadable firmware which I hope to share with the Cheap Yellow Display community.

If you can think of any additional features you would like to see integrated, please drop me an email using the address in my HPR profile.

I may also work with a friend to attempt to 3d print a case for the entire contraption, and I will be sure to record additional episodes sharing the process.

I have learned so much throughout this project, about the CYD, ESP32, GUIs, Claude Code, GitHub, and most of all, about myself.

Does using AI to develop this code make me a fraud? It still feels like it in some ways.

Does it make me more productive? ABSOLUTELY! I made consistent forward progress when I only had 30-60 minutes each day to work on it, and everything discussed in this episode was completed in less than a week. If I had been able to work on it for a few hours uninterrupted, it may have only taken me 3-5 hours.

Does it empower and inspire me to do more projects like this? 100% I feel like I had support working with me the whole way. I was less stressed overall, and it had less of an impact on the amount of and quality of time I spent with my family.

I will be wrapping up this series soon, without any more 6 month gaps, I hope.

Until next time...

Provide feedback on this episode.

01 Introduction

In this episode I will describe techniques for downloading podcasts using basic shell commands such as wget.

I will illustrate this using a bash script that can be used to download HPR podcasts.

Even if you do not have any interest in downloading your podcasts using this method, you may find some of the methods useful or interesting.

It is the principles that are discussed here that are important, rather than the implementation.

02

I realize that there are already a number of different podcast download programs available, including at least one written in bash.

However, you may feel that none of these suit how you wish to do things and want to create your own system tailored to your specific needs.

If so, then I hope the following is of some use to you.

If not, then you may still find some of the things discussed here to still be of interest.

Some of the subjects I cover include

wget to a user defined file name.

parsing xml with xmllint.

using inotifywait to trigger an action when a file is created or modified.

using notify-send to send a message to the notification area.

and

a way of allowing a cron job to send a message to the user interface.

03 Background

There has been an ongoing discussion in comments to some HPR episodes about problems downloading HPR podcast episodes.

Apparently some people have been experiencing problems with the way the episode URLs are structured.

04

I am afraid that I don't fully understand the nature of these problems, so I won't be addressing that problem directly.

Instead, I will present a bash script that I have written which can be used to download HPR podcasts.

This bash script can be run using cron to automatically fetch new HPR podcasts and save them to a designated directory.

This is a simplified version of a script that I have used for years to download HPR and other podcasts.

05

I won't try to read the full bash script out in this podcast, as that would be a bit dull to listen to.

I will instead describe what each section does and why I chose to do things that way.

Perhaps other people can offer suggestions of better ways to do things.

I will post the full bash script in the show notes.

06 Fetching Podcasts

The standard way of distributing podcasts is to publish an RSS feed containing URL links to the audio files.

RSS is a very long established and widely supported mechanism for this and other purposes.

An RSS feed is basically an XML document which can be accessed over HTTP.

These URLs contained in the RSS XML document can then be used to download the actual audio files, such as MP3 or OGG files.

07

Basically what we need to do is the following

• Download the RSS XML document.

• Extract the URL links to the audio files.

• Compare the list of these links to a previously saved list to see which ones are new and which ones are ones that we previously downloaded.

08

• Make a list of the new URLs.

• Go through this list of new URLs and download each of the new audio files.

• Check to see that we actually received the new audio file.

• Add the URLs of the files we successfully downloaded to our saved list of podcast URLs

09

In addition to this, we would like to have the above happen automatically in the background without our having to take any action on our own.

We may wish to receive a notification of when a new podcast has arrived however.

We would probably also wish to receive notification of any errors or failures.

10 Fetching Podcasts - The Preliminaries

Our desire to be able to run the script automatically imposes some requirements on our solution.

To schedule the script we will use cron.

Cron is a Linux facility to run scripts on a schedule.

11

One of the side effects of using cron however is that we need to specify the full path to the locations where we intend to keep any data files, plus also the full path to where we intend to put the downloaded podcasts.

12

So the first thing we need to do in our script is to specify a number of different values for things like file location, the URL for the HPR RSS feed, and several other things as well.

I will skip over the details of these, although I may make reference to them later.

13 Get the RSS Data

The first thing of real substance to do is to fetch the current RSS feed data.

I have put this in a bash function called getrssurldata

The contents of this function are a one liner, but with a number of elements chained together through pipes.

14 Downloading the RSS XML Document

• First we use wget, which is a standard command on most Linux distros.

• We specify four things.

• First we set a timeout. I have chosen 20 seconds.

• Next we set the retry limit. I have chosen 3.

15

• Then we specify that the output of wget is sent to stdout rather than saved as a file.

• This is done by using the -O option followed by a space and then a dash.

• The O option is usually used to specify a file to save the output to, but when used with a dash causes output to go to stdout.

• Then we specify the URL of the HPR RSS feed.

16 Contents of the XML Document

This gives us the HPR RSS XML document.

There are about 5,000 lines in this RSS document.

Most of those lines are the show notes which are also included in the feed.

17 Extracting the Podcast Episode URLs

There are only 10 lines of the document that contain information that we are interested in however.

These lines are enclosed in "enclosure" XML tags.

We just need to find those lines and separate out the URLs

18 Standard Command Line Tools

There are two ways that we can do this.

One is to use a combination of grep, sed, and cut.

Grep can find the lines containing the enclosure tags.

Sed and cut can extract the URL from the surrounding extraneous data.

19

However, this method does not discriminate between real enclosure tags in the data portion of the RSS feed and enclosure tags in the show notes which are included in the feed from episodes such as this one.

This may be an acceptable problem in practical terms, but we can do better.

20 Using an XML Parser

The other method is to actually parse the XML document.

there are at least two command line XML parsers that I am aware of.

These are "xmllint", and "xlmstarlet".

I have used xmllint in this example.

I have not used xmlstarlet, so I can't offer any comment on how easy or difficult to use it is.

21

I won't give a detailed explanation of all the things that xmllint can do.

It has many features, most of which, as the name suggests, have to do with finding formatting problems with the XML itself.

Describing everything it can do would be at least one episode in itself.

I will instead just give the particular command used and explain each element of it.

22

In this example assume that we are piping the output of wget directly into xmllint.

The complete command is

xmllint --xpath "//channel/item/enclosure/@url" - | cut -d'"' -f2

23

In this example,

xmllint is the name of the command.

--xpath tells it to parse the document according to the string which follows.

"//channel/item/enclosure/@url" tells it to find a series of tags in the hierarchy of channel, followed by item, followed by enclosure, and then extract the url attribute from the enclosure tag.

The "-" which follows tells it to look for input from stdin rather than from a file.

24

The result is a string which has the url attribute name, an equal sign, and the URL that we want enclosed in quotes.

To get just the URL itself, we pipe the output from xmllint into cut, using the doublequote characters as delimiters.

We then save the result in a temporary file.

25 Finding the New Episodes

Next we wish to find the new podcast episodes.

Each HPR episode is identified by a unique URL.

This means that if we save the URLs of episodes that we have already downloaded, we just have to look for the URLs that do not appear in this saved list.

https://hub.hackerpublicradio.org/ccdn.php?filename=/eps/hpr4659/hpr4659.mp3

26

The easiest way to do this is to take our two lists of URLs, sort each into temporary files, and then compare the sorted URLs using the "comm" command.

27

This is simple, but has a drawback.

Some podcasts occasionally change distributors.

When they do this, the old podcasts are re-published with new URLs and you end up downloading a lot of old episodes over again.

28

With HPR we could get around this by extracting just the file name and looking for that instead of the full URL.

I will however leave that problem as an exercise for the student and just accept that if the URL format changes we may end up downloading old episodes over again.

Since the feed has a maximum of only 10 episodes in it however, that isn't really that big of a problem.

It would be more of a problem with podcasts which have very large numbers of episodes in their feed, but the solutions to those will be feed specific.

29 Downloading the New Podcasts

We should now have a list of URLs for the new podcasts we do not already have.

Typically this should be only one file, but there could be several, or even as many as 10, if we have not turned on our computer in a while.

Therefore, we need to iterate through the file of new podcast URLs and download each one.

30

Before we do that however, we should check to see if there is in fact anything new to download.

To do this, simply use "wc -l" to count the number of lines in the list of new URLs and save the resulting number.

31

If this number is zero, there is nothing to download, we can skip the download step.

As an additional check, we should see if the number of downloads exceeds some threshold value that we wish to set.

This is not a major problem with HPR, but some podcasts have hundreds of files in their RSS feed rather than just the most recent ones.

If we do exceed our download limit, then we need to log an error and skip downloading.

32

Assuming there are no problems so far however, the first thing we need to do is to extract the name of the audio file from the URL.

We can do that using the "basename" command.

We will use this to specify the name that we use when we save the audio file.

33

HPR has a very well formed file name.

Some podcasts do not however, and for those you would need to construct some sort of suitable name either using information found in the URL or simply creating a name using a time stamp.

34

Next we download the audio file using wget.

This is similar to how we downloaded the RSS feed, but with a few changes.

One is that I have increased the timeout to 90 seconds.

This may not have been necessary, but seemed like a good idea.

35

The next is that when specifying the output file name using -O, we use the file name we extracted from the URL.

The third is that we specify a destination directory using the -P option.

36

After wget has finished, including any retries that it had to do, we next check that the expected new file is both present and not empty.

We did this using an "if" statement with the "-s" option.

If the file was found and not zero, then we add that URL to a temporary list of downloaded URLs.

37

If the file was not present, or was zero length, we output an error message to an error log.

I will come back to this point later.

38

Next, if there is more that one podcast to download we sleep for 3 seconds.

While not strictly necessary, it is considered to be "polite" to not hammer a server repeatedly, but rather to put a small delay between file downloads..

39

After we have downloaded all the audio files in our list, we can add the list of URLs for the files downloaded to the permanent list.

While we are at it, we should use "tail" to trim the permanent log to keep it from growing indefinitely.

This limit should be several times bigger than the number of files in the RSS feed.

In this case I selected 50.

40

Finally we write any errors to the permanent error log, and also write these same errors to another file used to signal errors for display to the user.

We have now successfully downloaded at least one HPR podcast.

41 Notify the User of Events

It would be convenient to be informed of new podcast downloads when they occur, and also be notified of any errors.

One of the limitations of cron jobs is that they cannot access the user interface.

This means that we cannot readily send a message directly to the notification system to inform the user of the presence of new podcasts or of errors.

42 inotifywait

The solution to this is to use "inotifywait" to monitor particular files and directories for changes.

The man page for inotifywait states the following -

43

inotifywait efficiently waits for changes to files using Linux's inotify(7) interface. It is suitable for waiting for changes to files from shell scripts. It can either exit once an event occurs, or continually execute and output events as they occur.

End of quote.

44

In many Linux distros, inotifywait is provided by the "inotify-tools" package.

I won't go over all the features of inotifywait.

Instead, I will just describe how to use it for our purposes here.

45 inotifywait Modes

I should point out first though that inotifywait operates in two different modes.

In the normal default mode, it exits after being triggered by an event and must be re-established again in order to resume monitoring.

In monitor mode, which is enabled by using the "-m" option, it runs indefinitely, responding to events.

I will use the default mode here.

46

The man page for inotifywait provides a simple example that we could copy and modify for our purposes.

A great many examples that you will find are based on this example.

However, it doesn't quite do what we want, so we need to change a few things.

47 podfetchnotify

The first shell script is one which monitors for the arrival of new podcasts and sends a notification to the user.

I will call this "podfetchnotify".

The complete scripts are in the show notes, I will just provide a brief description here.

48 Setting Up Event Watches Using inotifywait

The script is enclosed in a while loop which run indefinitely.

In the first line inside the while loop, we call inotifywait.

inotifywait will then block until the event it is told to look for occurs.

In short, execution of the script will wait there until an event occurs.

49

The names of the events are listed in the man file.

In this case we are looking for "modify", "create", and "moved_to".

Each of these does pretty much as you would expect, reacting to modifying an existing file, creating a new file, or moving a file to that directory.

50 Problems When Testing Using Text Editors

I should point out that if you are testing a script which uses inotifywait, then modifying a file with a text editor may not produce the results that you may think it would.

Instead it treats this as a new file with the same name, with the original file being erased.

Since inotifywait attaches itself to the inode rather than the filename, it sees the file that the text editor changed as being a new file.

If you wish to test this realistically, then use "echo" to overwrite the file by using I/O redirection.

51 Capturing Output

In my example I capture the output from standard out into a variable, but I don't do anything with it.

If you wish to for example display the name of the newly downloaded podcast file, then use the --format option along with an appropriate formatting code.

There are details about this in the man page.

On the next line we capture the exit code using "$?"

52 Responding to Exit Codes

If the exit code was zero, then a monitored event was triggered and there should a new podcast in the directory.

In this case we display a message indicating that a new podcast has arrived.

I will describe how to send notifications shortly.

If the exit code was not zero, then an error occurred.

An example of such an error would be if the directory were not present when monitoring was started.

In this case we display a message indicating that a fatal error has occurred and then exit.

53 Delay for More Podcasts

Finally, we use "sleep" to wait for some arbitrary period of time to prevent notifications from being triggered multiple times if several podcasts were being downloaded in succession.

In this case I chose to wait for 60 seconds.

54

We have now completed the process and can return to the top of the loop and resume waiting using inotifywait.

55 Sending Notifications to the User

I mentioned above about sending notification messages to the user.

In the Gnome desktop, notification messages appear from the centre of the top bar in a list.

Other desktops or operating systems may have something similar.

56

To send a notification message to the notification area, you use the "notify-send" command.

Simply follow notify-send with a quoted string and it will be displayed in the notification area.

57 podfetcherrornotify

The second shell script is one which notifies the user of errors.

I will call this "podfetcherrornotify".

With this shell script we set up a watch on a file which contains any error messages from podfetch.

This script is very similar to podfetchnotify.

58

The exceptions are

With inotifywait we only monitor for "modify".

There is no sleep command at the end of the loop.

Instead we sleep for a few seconds just after getting the exit code from inotifywait.

This helps prevent problems caused by race conditions.

59

Next we check the inotifywait exit code.

If it was zero, then we read the error report file and send a notification message to the user containing that error message.

60

If it was not zero, then we check to make sure that the directory that should contain the error log exists.

If it does not exist, then we send a notification message to that effect to the user and terminate the script.

61

If the directory exists, then we check to see if the error message file used for signalling exists.

If the file does not exist, then we create it.

62

One of the reasons for an inotifywait error is that if the file that it is told to monitor does not exist, it cannot set up a watch condition.

By creating the file we correct the cause of the error and allow inotifywait to operate normally.

63

Finally we increment an error counter and check to see if the limit is exceeded.

If there are excessive errors, then send a notification message to the user and exit.

The reason for this is to give the user an indication that the error notifications are not working for some reason and there may be a problem that needs looking into.

64

The error counter is reset every time the inotifywait exit status is ok, so occasional unexpected glitches should be something that is ignored.

Of course podcast fetching errors are something that will probably happen only rarely if at all, so this final step may be seen as an unnecessary embellishment.

65 Installing the Scripts

Next I will describe how to install and prepare the scripts to run.

We need to perform the following steps.

66

• First, we need to create a directory to hold the scripts and their associated data files.

• Next we need to create a directory to hold the downloaded podcasts.

• Then we must copy the scripts to these directories and make them executable.

• Then, we must edit the scripts to have the file path in the script match the locations of the new directories that we created.

67

• Then we need to install xmllint, or alternatively modify the download script to comment out the use of xmllint and enable the alternative method using grep and sed instead.

• Then we need to run each script manually from the command line to check for errors.

• If podfetch ran correctly, it should download the most recent 10 podcasts during this test.

68 Adding podfetch to the Crontab

The above describes how to run the scripts manually.

In order to fetch podcasts automatically, we need to add the podfetch script to the cron schedule.

To do this, open a terminal.

69

Type "crontab -e", and then press return.

A text editor should open up containing the crontab file.

On Ubuntu, this editor is GNU nano.

Enter the appropriate cron parameters.

I will provide an example here for running it 12 minutes past the hour every three hours.

70

12 */3 * * * /home/username/pathtofiles/podfetch.sh

71

I won't explain cron in detail here.

The example that I have just given should be good enough for most people.

The "*/3" parameter will cause it to run every three hours.

The "12" parameter will cause it to run 12 minutes past the hour when it does run.

72

Checking every three hours should be good enough for most people, but you can adjust that as you see fit.

I would recommend however that you don't check more frequently than once per hour.

Checking more frequently than necessary puts extra load on the distribution servers.

It is very unlikely that you really do need each new episode the moment it is available.

73

I would also recommend changing the "12" parameter to some other random minute value.

I would suggest avoiding on the hour or on the half hour, as a lot of other people are probably checking at those times, and it would be better to spread the load out more evenly over time.

74

The file path parameter should of course match the actual path to wherever you have located the script, including the correct user name.

75 Making the Notification Scripts Start Automatically

The two notification scripts can be made to start automatically.

The exact method to do this may vary according to distribution or desktop.

76

On Ubuntu this is done using the Startup Applications Preferences GUI program, which should come already installed.

77

I won't go into details on this here, it should be fairly self evident how to use it once you see it.

What this program does is to create ".desktop" files in the ".config/autostart" directory in your home directory.

78

These ".desktop" files are all run automatically on start up.

Once you have added the notification scripts, you will need to log out and then log back in to make them active.

79 Conclusion

I this episode I explained how to write a set of simple shell scripts to automatically download each new episode of HPR as it comes out and to notify you of its arrival.

80

The download script described here is tailored specifically for use with HPR only.

However, it was derived from a larger script that downloaded other podcasts as well, based on information read in from a text file.

If you are feeling ambitious, you can add those features back into this to handle all of the podcasts that you listen to.

81

In a comment to another episode of HPR I had said that I would cover ID3 tags in MP3 files, but this episode is long enough now, so I will leave that subject for later.

I look forward to seeing you again later on another episode of Hack Public Radio.

# ======================================================================

podfetchdownloader

#!/bin/bash

# Fetch pending HPR podcasts listed in the HPR RSS feed.

# 8-Jun-2026

# Licensed under GPLv3 or later.

# ======================================================================

# Today's date and time as YYYYMMDDHHMMSS.

podttimestamp=$( date +"%Y%m%d%H%M%S" )

# The absolute path to the script. This is necessary when running it

# using a cron job.

podpath="/home/me/Apps/hprfetch"

# This is the absolute path to where to store the podcast files.

podfilepath="/home/me/Music/Podcasts/HPR"

# Create the full path names here for all the text files used.

podcastsfetched="$podpath/podcastsfetched.txt"

poderrorslog="$podpath/poderrorslog.txt"

poderrorsreport="$podpath/poderrorsreport.txt"

tmpoldurlssorted="$podpath/tmpoldurlssorted.txt"

tmppodsnew="$podpath/tmppodsnew.txt"

tmppodstodownload="$podpath/tmppodstodownload.txt"

tmppodserrors="$podpath/tmppodserrors.txt"

tmppodcastsfetched="$podpath/tmppodcastsfetched.txt"

tmplog="$podpath/tmplog.txt"

# The URL for the HPR RSS feed.

PodURL="http://hackerpublicradio.org/hpr_rss.php"

# Limit on number of podcasts to download.

DownloadLimit=11

# Name of the podcast.

PodName="Hacker Public Radio"

# ======================================================================

# Check if the required paths exist.

# If this path does not exist, cannot log the error.

if [[ ! -d "$podpath/" ]]; then

echo "$podttimestamp Error - Could not find $podfilepath."

exit 1

fi

# Where to store the podcast file fetched.

if [[ ! -d "$podfilepath/" ]]; then

echo "$podttimestamp Error - Could not find $podfilepath." >> $tmppodserrors

# Copy the errors log from the temporary errors file to the permanent files.

LogErrors

exit 1

fi

# ======================================================================

# Check if the podcast log exists. We read it before we write to it,

# so it must exist or we will hang on it not being present.

if [[ ! -e $podcastsfetched ]]; then

touch $podcastsfetched

fi

# ======================================================================

# Delete the specified files if they exist.

# This accepts multiple file names in a variable number of parameters.

CleanupFiles ()

{

# $@ accepts multiple parameters.

for f in "$@"; do

# Check if the file exists.

if [ -e "$f" ]; then

rm "$f"

fi

done

}

# ======================================================================

# Copy the errors log from the temporary errors file to the permanent files.

LogErrors () {

if [ -e $tmppodserrors ]; then

# The permanent log.

cat $tmppodserrors >> $poderrorslog

# This file is monitored for display by other scripts.

cat $tmppodserrors > $poderrorsreport

fi

}

# ======================================================================

# Get the URL data from an RSS feed

GetRSSURLData () {

wget --timeout=20 --tries=3 -O - "$PodURL" \

| xmllint --xpath "//channel/item/enclosure/@url" - | cut -d'"' -f2 \

| sort > $tmppodsnew

# This is an alternate method that does not use xmllint.

# However, it is not as robust. If someone were to include the

# first grep search pattern in their show notes, then it would

# look for that as a valid tag and output the following text

# as a URL.

#wget --timeout=20 --tries=3 -O - "$PodURL" | grep "<enclosure url=" \

# | sed -n 's/^.*enclosure//p' | sed -n 's/^.*url=//p' \

# | cut -d'"' -f2 | sort > $tmppodsnew

}

# ======================================================================

# Find which podcasts we do not already have.

FindNewPodcasts () {

cat $podcastsfetched | sort > $tmpoldurlssorted

comm -13 $tmpoldurlssorted $tmppodsnew > $tmppodstodownload

rm $tmpoldurlssorted

}

# ======================================================================

# Download the podcasts.

DownloadPodcasts() {

# Clear out previous temporary list of downloaded podcasts.

true > $tmppodcastsfetched

for i in $( cat $tmppodstodownload )

do

# Extract the file name from the URL.

fname=$( basename $i )

outputpodname="$podfilepath/$fname"

# Download the file.

wget --timeout=90 --tries=3 -P $podfilepath $i -O "$outputpodname"

# Check if the file exists and is not empty.

if [[ -s "$outputpodname" ]]; then

echo $i >> $tmppodcastsfetched

else

echo "$podttimestamp Error - $outputpodname was not found or is empty." >> $tmppodserrors

fi

# Delay a reasonable length of time between multiple downloads.

if (( $PodCount > 1 )); then

sleep 3

fi

done

# Add the list of files downloaded to the log.

# Check if the list exists and is not empty.

if [ -s $tmppodcastsfetched ]; then

cat $tmppodcastsfetched >> $podcastsfetched

# Trim the log file to keep it from growing indefinitely.

tail -n50 $podcastsfetched > $tmplog

mv $tmplog $podcastsfetched

fi

# Remove the tmp file now that we are done with it.

rm $tmppodcastsfetched

}

# ======================================================================

# Clean up any left over files.

CleanupFiles "$tmppodsnew" "$tmppodstodownload" "$tmppodserrors" "$tmppodcastsfetched"

# Get the RSS data.

GetRSSURLData

# Find which podcasts are new.

FindNewPodcasts

# Count how many new podcasts there are.

PodCount=$( cat $tmppodstodownload | wc -l )

# If no podcasts to download, skip this.

# If too many podcasts for this feed, then log an error and skip.

# This error will keep repeating until something is done about it.

if (( $PodCount > 0 )); then

if (( $PodCount > $DownloadLimit )); then

echo "$podttimestamp Too many podcasts for $PodName : $PodCount." >> $tmppodserrors

else

# Download the podcasts listed in the temp file.

DownloadPodcasts

fi

fi

# ======================================================================

# Copy the errors log from the temporary errors file to the permanent files.

LogErrors

# Clean up temp files.

CleanupFiles "$tmppodsnew" "$tmppodstodownload" "$tmppodserrors" "$tmppodcastsfetched"

# ======================================================================

END OF FIRST SHELL SCRIPT

START OF SECOND SHELL SCRIPT

podfetchnotify

#!/bin/bash

# Part of Podfetch.

# This monitors for new files appearing in the new podcasts directory.

# This should be run as a background task.

# Install it using the "Startup Applications" utility in Ubuntu.

# ======================================================================

# Path where new podcasts are to be stored.

podfilepath="/home/me/Music/Podcasts/HPR"

# ======================================================================

# Wait for the podcast directory to be modified.

while true; do

# Check for new files.

errmsg=$( inotifywait -e modify -e create -e moved_to $podfilepath )

result=$?

# Check if exited due to new podcast, or if some error.

if (( result == 0 )); then

# Success, signal new podcast.

notify-send "New HPR podcast available."

else

# Check to make sure the directory exists.

# If it doesn't exist, there isn't much we can do to fix it.

if [ ! -e "$poderrorspath" ]; then

notify-send "Podfetch error: Podcast directory not found $poderrorspath"

exit 1

fi

fi

# Wait a bit so that multiple new files don't keep re-triggering the notification.

sleep 60

done

# ======================================================================

END OF SECOND SHELL SCRIPT

START OF THIRD SHELL SCRIPT

podfetcherror

Created Tuesday 23 June 2026

#!/bin/bash

# Part of Podfetch.

# This monitors the Podfetch error reporting file for new errors.

# This should be run as a background task.

# Install it using the "Startup Applications" utility in Ubuntu.

# ======================================================================

# Where the Podfetch program error report file is located.

poderrorspath="/home/me/Apps/hprfetch"

# The full path and file name.

poderrorsreport="$poderrorspath/poderrorsreport.txt"

# ======================================================================

# Error counter.

errcount=0

# Wait for the poderrorsreport file to be modified.

while true; do

errmsg=$( inotifywait -e modify $poderrorsreport )

result=$?

# Wait a bit to ensure that writing to the file is complete.

sleep 3

if (( result == 0 )); then

# Get the latest error message.

# Cut out the date stamp at the start of the line and take the rest.

poderr=$( tail -n $poderrorsreport | cut -d" " -f2- )

notify-send "Podfetch error: $poderr"

# Reset the error counter every time there is a successful result.

errcount=0

else

# Check to make sure the directory exists.

if [ ! -e "$poderrorspath" ]; then

notify-send "Podfetch error: error report path not found $poderrorspath"

exit 1

fi

# Check if the file we are trying to monitor exists.

# If not, then create an empty file for error signaling.

if [ ! -e "$poderrorsreport" ]; then

echo > $poderrorsreport

fi

# Increment the error counter.

count=$(( count + 1 ))

if (( count > 3 )); then

notify-send "Podfetch error: Excessive unknown errors, exiting."

exit 1

fi

fi

done

# ======================================================================

Provide feedback on this episode.

Sponsors

Support for this show comes from ThreatLocker®. ThreatLocker® is a Zero Trust Endpoint Protection Platform that strengthens your infrastructure from the ground up. With ThreatLocker® Allowlisting and Ringfencing™, you gain a more secure approach to blocking exploits of known and unknown vulnerabilities. ThreatLocker® provides Zero Trust control at the kernel level that enables you to allow everything you need and block everything else, including ransomware! Learn more at www.threatlocker.com.

This show is sponsored by Maze. Maze uses AI agents to triage and remediate cloud vulnerabilities by figuring out what’s actually exploitable, not just what’s theoretically risky. They remove the noise, prioritize vulns that matter, and manage remediation, so your team stops wasting time on meaningless vulns. Visit MazeHQ.com/darknet for more information.

This show is sponsored by Sentry.IO. Sentry wants to help you monitor your environment for problems. They do error tracking, stack tracing, debugging, all so that your developers can diagnose, fix, and optimize the performance of their code. This makes it so developers ship more reliable code faster. Learn more at sentry.io.

View all active sponsors.

Sources

Full list of sources on the show page: https://darknetdiaries.com/episode/177/

ether

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.

I frequently find myself reaching for the cut utility when writing scripts to extract one piece of data from a line, or to select specific fields from a log file. While I am familiar with its counterpart, paste , I don't employ it very often because I don't typically need its functionality.

This perhaps has to do with the fact that I rarely work with text files containing lists. For shorter lists, I usually end up using a spreadsheet and for larger ones, a relational database. Both are valuable tools with their own strengths and weaknesses, but it is good to also know about standard utilities for working with lists. After uploading UNIX Curio #8 ( HPR episode 4657 ), I felt like maybe I had been too dismissive of the comm utility in that episode and should talk more about tools that are useful when managing lists.

I don't frequently find myself using paste ¹ , but can explain how it works. Briefly, it is a rough opposite of cut —when given multiple files as arguments, it assembles the first line from each one separated by tabs, then the second line, and so on. Instead of tabs, a different delimiter can be chosen with the -d option. Another option is -s , which swaps rows and columns so that the contents of each named file would appear on one line. While paste itself doesn't qualify as a UNIX Curio in my opinion, there is one feature that does: a hyphen can be given as an argument multiple times. In this special case, the output is taken line by line from standard input, but is spread across as many columns as there are hyphens.

Example of using paste to turn the output of ls into columns. Because these columns are separated by tabs, they don't necessarily line up when a filename is eight or more characters long. The -1 is not required for the second ls command since that behavior is implied when output isn't going to a terminal. The -C option to ls usually gives nicer-looking output on a terminal—also, it lists in ascending order down by column. (Most implementations default to -C when output goes to a terminal.) If you want items ascending along rows like the paste example does, try ls -x instead.

$ ls -1 /proc/net
anycast6
arp
bnep
connector
dev
dev_mcast
dev_snmp6
fib_trie
fib_triestat
hci
icmp
icmp6
if_inet6
igmp
igmp6
ip6_flowlabel
ip6_mr_cache
ip6_mr_vif
ip_mr_cache
ip_mr_vif
ip_tables_matches
ip_tables_names
[...35 more entries not shown...]
$ ls /proc/net | paste - - - -
anycast6        arp     bnep    connector
dev     dev_mcast       dev_snmp6       fib_trie
fib_triestat    hci     icmp    icmp6
if_inet6        igmp    igmp6   ip6_flowlabel
ip6_mr_cache    ip6_mr_vif      ip_mr_cache     ip_mr_vif
ip_tables_matches       ip_tables_names ip_tables_targets       ipv6_route
l2cap   mcfilter        mcfilter6       netfilter
netlink netstat packet  protocols
psched  ptype   raw     raw6
rfcomm  route   rt6_stats       rt_acct
rt_cache        sco     snmp    snmp6
sockstat        sockstat6       softnet_stat    stat
tcp     tcp6    udp     udp6
udplite udplite6        unix    wireless
xfrm_stat
$ ls -C /proc/net
anycast6      if_inet6           l2cap      rfcomm        tcp
arp           igmp               mcfilter   route         tcp6
bnep          igmp6              mcfilter6  rt6_stats     udp
connector     ip6_flowlabel      netfilter  rt_acct       udp6
dev           ip6_mr_cache       netlink    rt_cache      udplite
dev_mcast     ip6_mr_vif         netstat    sco           udplite6
dev_snmp6     ip_mr_cache        packet     snmp          unix
fib_trie      ip_mr_vif          protocols  snmp6         wireless
fib_triestat  ip_tables_matches  psched     sockstat      xfrm_stat
hci           ip_tables_names    ptype      sockstat6
icmp          ip_tables_targets  raw        softnet_stat
icmp6         ipv6_route         raw6       stat
$ ls -x /proc/net
anycast6           arp              bnep               connector     dev
dev_mcast          dev_snmp6        fib_trie           fib_triestat  hci
icmp               icmp6            if_inet6           igmp          igmp6
ip6_flowlabel      ip6_mr_cache     ip6_mr_vif         ip_mr_cache   ip_mr_vif
ip_tables_matches  ip_tables_names  ip_tables_targets  ipv6_route    l2cap
mcfilter           mcfilter6        netfilter          netlink       netstat
packet             protocols        psched             ptype         raw
raw6               rfcomm           route              rt6_stats     rt_acct
rt_cache           sco              snmp               snmp6         sockstat
sockstat6          softnet_stat     stat               tcp           tcp6
udp                udp6             udplite            udplite6      unix
wireless           xfrm_stat

The paste command has limitations—the files you give it must all be already arranged in the same order, and if any file is missing a value, it must have a blank line so that subsequent lines will match up correctly. The files do not necessarily have to be sorted alphabetically, but whatever order they are in has to be the same. Check out HPR episodes 962 and 4201 for some more background on the paste utility.

Example of using paste with files where some values are empty. Bob works from home so doesn't have an office assigned, and the laboratory Carol works in doesn't have a phone. This relies on the fact that the same line number in every file relates to the same person/entry.

$ cat names
Alice
Bob
Carol
Dave
$ cat offices
203

Lab6A
117
$ cat phones
+1 212-555-1234
+1 919-555-2345

+1 212-555-1278
$ paste names offices phones
Alice   203     +1 212-555-1234
Bob             +1 919-555-2345
Carol   Lab6A
Dave    117     +1 212-555-1278

Our second UNIX Curio for today is a utility called join ² , which has a bit more sophistication. It operates on two files, which can have multiple columns, and combines them using the join field. By default, the first column/field in each file is the join field, and only entries that exist in both files are printed. The -1 and -2 options can be used to join on a different field, and -o selects specific fields to be output. To make it so lines with missing entries also appear, you need to use the -a option, but an actual empty string with separator won't be printed unless -o is also present and includes the field.

The default field separator character is one or more "blanks" in the current locale—for the POSIX locale, this means a space or a horizontal tab. The -t option selects a different character and also removes the treatment of multiple occurrences as a single separator, making it possible to have an empty field in one or both of the files. By default, a single space is used to separate fields in the output. If -t is given, the same character is used for separating fields in both input and output. You would need to pipe output through another tool like tr if you wanted to have a different separator in the output.

The join utility might be an improvement over paste in some cases, since the join field makes it a little easier to identify which entries match up across files. It is limited to operating only on two files (one of which can be standard input), so combining more than that requires either creating temporary intermediate files or chaining together join commands in a pipeline. Another requirement is that all files must already be sorted in the current locale.

Example showing how join can be used with two tab-separated lists. The LC_ALL assignment forces join to sort using the C (POSIX) locale instead of whatever might be set in your environment. The "@" on the header line has no special meaning; it is just there to make sure it sorts before any letters or numbers (in the C locale; it might not in other locales). Note that if -t were not specified, plist would be treated as having three fields because of the space separating the country code from the rest of the phone number.

$ export tab="$(printf '\t')" #To more easily use tab characters below
$ cat olist
@Name   Office
Alice   203
Carol   Lab6A
Dave    117
$ cat plist
@Name   Phone
Alice   +1 212-555-1234
Bob     +1 919-555-2345
Dave    +1 212-555-1278
$ LC_ALL=C join -t "$tab" olist plist
@Name   Office  Phone
Alice   203     +1 212-555-1234
Dave    117     +1 212-555-1278
$ LC_ALL=C join -t "$tab" -a 1 -a 2 olist plist
@Name   Office  Phone
Alice   203     +1 212-555-1234
Bob     +1 919-555-2345
Carol   Lab6A
Dave    117     +1 212-555-1278
$ #By default, join acts as if empty fields don't exist; use -o to include
$ LC_ALL=C join -t "$tab" -a 1 -a 2 -o 0,1.2,2.2 olist plist
@Name   Office  Phone
Alice   203     +1 212-555-1234
Bob             +1 919-555-2345
Carol   Lab6A
Dave    117     +1 212-555-1278
$ #The -e option sets a placeholder to use for empty fields
$ LC_ALL=C join -t "$tab" -e "(none)" -a 1 -a 2 -o 0,1.2,2.2 olist plist
@Name   Office  Phone
Alice   203     +1 212-555-1234
Bob     (none)  +1 919-555-2345
Carol   Lab6A   (none)
Dave    117     +1 212-555-1278

The brief description for join is "relational database operator"—I won't dispute that, but in my view it offers far fewer capabilities than people would expect from today's relational databases. I would imagine that when most people think of those they have Structured Query Language (SQL) in mind, which offers a lot more flexibility and functions to operate on data. However, I can see how join could be suitable for simple operations.

Our last UNIX Curio for today relates to the sort utility ³ . While, as you might expect, it is well-known for its ability to sort data, it has another feature that is more obscure. When used with the -m option, instead of sorting the files given as arguments, it merges them together. All of the files are expected to already be sorted—once combined, the list that is output will also be sorted. The order in which the files are named does not matter; it is not required for the contents of the first file to start before the second, just that both are sorted.

$ cat women
Alice
Carol
$ cat men
Bob
Dave
$ sort -m men women
Alice
Bob
Carol
Dave

Imagine that you organize an annual event and have a separate pre-sorted list of attendees' e-mail addresses for each of the past three years. You are planning this year's event and want to send out an announcement to all of these people, as they will probably be interested. The command sort -m -u 2023list 2024list 2025list would spit out a combined list that you can use for your e-mail blast. Because it is likely that some people would have attended in more than one year, I included the -u option—it removes any duplicate entries.

It is probably no surprise that the sort utility appeared early on—it was in 1971's First Edition UNIX, though it didn't gain the merging functionality until Fifth Edition ⁴ in 1973. What did come as a shock to me is that both cut and paste didn't show up until 1980 with System III ⁵ , and were actually preceded by join , which was in Seventh Edition UNIX ⁶ from 1979. I assumed that at least cut would have been around far earlier, given its usefulness and how firmly established it is, but I suppose it just seems to have been with us forever.

As mentioned, I don't typically manage data as text files containing lists, and I probably won't start using the join utility or these features of paste and sort very much. But it is still useful to know that they exist and how they work. Hopefully this episode has taught you a bit about them.

References:

1. Paste specification https://pubs.opengroup.org/onlinepubs/9699919799/utilities/paste.html
2. Join specification https://pubs.opengroup.org/onlinepubs/9699919799/utilities/join.html
3. Sort specification https://pubs.opengroup.org/onlinepubs/9699919799/utilities/sort.html
4. A Research UNIX Reader: Fifth Edition sort manual page https://archive.org/details/a_research_unix_reader/page/n19/mode/1up
5. System III paste manual page https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/man/man1/paste.1
6. Seventh Edition UNIX join manual page https://man.cat-v.org/unix_7th/1/join

Provide feedback on this episode.

Show Notes

Episode Overview

• Operator kicks off the episode feeling under the weather but shares a quick tip for making perfect egg drop soup before diving into his main project: diagnosing why his front-door security camera stopped sending alerts and recording events. What follows is a live-debugging session covering network config, script logging, Windows permission hacks, NTP time drift, and firmware flashing.

Key Topics & Breakdown

• • Egg Drop Soup Hack: How to get that perfect ribbony texture by creating a boiling swirl before pouring in the eggs, plus broth-to-egg ratio tips.
  • Camera Setup & Network Config: Using static DHCP via MAC address binding on a UniFi Dream Machine (UDM) for local domain resolution instead of hardcoding IPs.
  • Python & Cron Automation: Running a custom Python script every 2 minutes to check for new recordings, parsing logs with grep -v , and navigating massive log files in vi .
  • Windows Troubleshooting Tangent: Deleting the stubborn Windows.old folder using the TrustedInstaller service hack ( ExecTI.exe ) instead of taking ownership manually.
  • Time Sync & Firmware Quirks: Discovering the camera's system clock was stuck in 2011/2026, causing missed events. Downloading firmware via a slow third-party link, renaming .bin to .zip , and extracting with 7-Zip.
  • Pre-Flash Backup Routine: Exporting camera configuration before upgrading, storing it in Google Drive for searchable documentation, and clearing old log/trigger files to reset the event pipeline.

️ Tools & Techniques Mentioned

• • crontab + Python scripts for automated monitoring
  • grep -v , cat , tail , and vi (line navigation with :1000 )
  • Obsidian for note-taking & AI assistant integration
  • Firefox/Playwright for headless browser testing
  • Turbo Download Manager & Bolt Media Downloader for multi-threaded/sniffing downloads
  • 7-Zip for archive extraction
  • Google Drive for searchable config backups

Resources & Links

• • Python API Script: Uniview IPC3628SR Recording Checker
  • Camera Model: IPC3628SR (Uniview Wyze ISP Warm Light Deterrent Network Camera)
  • TrustedInstaller Run-as Tool: ExecTI TrustedInstaller Runner

Quick Takeaways

1. 1. Always verify NTP/time sync on IoT cameras before troubleshooting missed events or alerts.
   2. Use grep -v "noise" to quickly filter out repetitive log entries when debugging automation scripts.
   3. Windows system folders can be stubborn; running commands as TrustedInstaller bypasses hidden file locks without manual ownership changes.
   4. Always export and back up device configs before flashing firmware, even if the upgrade seems straightforward.
   5. Third-party download links often use temporary tokens or .bin wrappers; renaming to .zip and verifying with 7-Zip can save headaches.

• Thanks for listening! Stay curious, keep your logs clean, and remember: defense in depth starts at home.

Example trusted installer hack

# Shhhh I can't IR ... Defender, ForcePoint, SMS Agent Host ...I just can't anymore ...

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Sense" /v Start /t reg_dword /d 4 /f"

sc start "TrustedInstaller"

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Fppsvc" /v Start /t reg_dword /d 4 /f"

sc start "TrustedInstaller"

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\CcmExec" /v Start /t reg_dword /d 4 /f"

sc start "TrustedInstaller"

sc config TrustedInstaller binPath= "Reg add "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinDefend" /v Start /t reg_dword /d 4 /f"

sc config TrustedInstaller binPath= "C:\Windows\servicing\TrustedInstaller.exe"

Provide feedback on this episode.

Lennart tells about the shortwave radio he had in the early 1980s and what he could hear on it, for example he marine band, amateur radio, CB and of course broadcast stations. The radio did not have a BFO to demodulate SSB stations, but using a second radio close by and using its local oscillator he could still make those signals intelligible.

Provide feedback on this episode.

Thrustmaster TMX Force Feedback Pro Black Xbox One / Xbox Series X/S / PC

https://www.ebay.com/itm/157554569422

Dayton Audio DAEX25 Audio Exciter Pair - Sound Exciter Pair Audio Transducer - 5 Watts RMS, 8 Ohms Impedance - 2 Pack - Turn Any Surface into a Speaker System

https://www.amazon.com/dp/B001EYEM8C

https://www.simhubdash.com/

https://www.accsetupcomparator.com/

https://www.iracing.com/ ( they have buy 2 years get discount during holiday but still can't afford it .. )

https://forza.net/horizon

DiRT Rally 2.0 GOTY

https://k4g.com/store?distribution[]=3&q=DiRT%20Rally%202.0%20GOTY&sort=price

CrewChiefV4

https://thecrewchief.org/

https://app.tracktitan.io/sessions/3ebbf63b-93de-4309-9a42-9311a745209d/20241228052252

SUMMARY.

User discusses sim racing challenges, costs, and setup tips.

IDEAS.

1. Sim racing requires time and investment.
2. I Racing is expensive with annual costs.
3. Set of Courses offers one-time payment.
4. Proper setup enhances sim racing experience.
5. Cable connections can complicate setup.
6. Upgrading hardware improves performance.
7. Arcade games like Forza offer casual play.
8. Realistic sim racing demands dedication.
9. License requirements vary between platforms.
10. Remote gaming systems save space.
11. Steering wheel upgrades improve smoothness.
12. Dedicated spaces optimize sim racing.
13. Balancing fun and realism is key.
14. Multi-launchers manage gaming platforms.
15. Hacked accounts may cause issues.
16. Monitoring hardware wear is important.
17. Shifting mechanisms enhance control.
18. Curved monitors improve immersion.
19. Time constraints affect sim racing participation.
20. Exploring multiple games adds variety.

RECOMMENDATIONS.

1. Consider Set of Courses for a one-time payment.
2. Invest in a proper setup for serious sim racing.
3. Use a multi-launcher for managing games.
4. Upgrade hardware for smoother performance.
5. Buy specific tracks and cars to avoid costs.
6. Opt for a dedicated space for sim racing.
7. Check for license requirements before purchasing.
8. Remote into gaming systems to save space.
9. Replace plastic parts with bearings for smoother operation.
10. Use a 7-speed shifter for better control.
11. Avoid hacked accounts for reliability.
12. Purchase multiple accounts for different players.
13. Focus on arcade games for casual play.
14. Prioritize a curved monitor for immersion.
15. Use a standing desk for accessibility.
16. Monitor cable connections to prevent setup issues.
17. Upgrade steering wheel components for better experience.
18. Balance fun and realism based on personal preference.
19. Consider time investment for sim racing.
20. Explore different racing games for variety.

Provide feedback on this episode.

Recording Kit

General

• Lanyard - HPR - Your Name
• HPR Business Cards - Your email address
• HPR Stickers
• Pen
• A6 Notebook
• Android Mobile phone OpenCamera

Zoom H2

• Zoom H2 Handy Recorder
• Set format to best and date format to ISO8601
• Replacement Batteries
• Micro USB Cable
• 2.5 mm headphones
• 3.5 mm Male to Male cable
• 3.5 mm earbuds

Backup 1

• Sansa Clip
• Rockbox
• Remove before flight key fob
• Set format to best and date format to ISO8601

Backup 2

• Android Mobile phone AudioRecorder
• Set format to best and date format to ISO8601

Provide feedback on this episode.

SUMMARY

The presenter outlines a practical cybersecurity workflow, covering ergonomic setups, browser isolation, virtual machine troubleshooting, AI-assisted scripting, and network tunneling methods utilized during active security assessments.

ONE-SENTENCE TAKEAWAY

Isolate browser environments, utilize automation scripts, and verify network paths before starting security tests to avoid workflow interruptions.

TOOLS

• Talon Voice – Open-source voice recognition software enabling hands-free computer control and command execution.
• Obsidian – Local-first markdown note-taking application supporting secure, AI-friendly knowledge management.
• AutoHotkey – Windows scripting utility for creating custom macros and remapping keyboard inputs.
• Chrome Debug Commands – Browser developer tools allowing direct inspection of extensions, cookies, and storage.
• Whisper Diarization – Audio processing script that separates speaker tracks and converts recordings to searchable text.
• Hyper-V / WSL – Microsoft virtualization platforms enabling isolated guest environments and Linux subsystem integration.
• OpenConnect / OpenVPN – Command-line tunneling clients used for establishing secure, split-tunnel network connections.
• Jamboree Framework – Portable PowerShell environment that dynamically provisions development tools without altering system paths.
• MOBA Portable – Feature-rich terminal emulator supporting static/dynamic tunnels, auto-reconnect, and embedded X-server capabilities.
• Nmap – Network discovery and security auditing tool utilized for comprehensive port scanning and service detection.

00:00:00 Ergonomic Workspace Configuration

Configures physical workstation elements to reduce strain during extended testing sessions. Proper alignment prevents repetitive stress injuries while maintaining focus on technical tasks.

• Monitor Positioning – Displays should align with eye level to maintain neutral neck posture; the speaker notes their curved 49-inch screen sits slightly high due to chair adjustments.
• Split Keyboard Layout – Utilizes a Freestyle 2 mechanical keyboard, allowing natural shoulder-width arm placement and reducing wrist deviation during prolonged typing.
• Postural Adaptation – Acknowledges that ergonomic equipment requires matching body alignment; elbow rests should sit between hip and shoulder height for optimal leverage.

01:45:00 Voice Control & Note Synchronization

Utilizes auditory input methods and localized knowledge bases to streamline documentation workflows. Separating secure work notes from casual observations prevents data contamination.

• Talon Voice Integration – Runs continuously to handle navigation, text entry, and application switching without manual keyboard interaction.
• Obsidian Migration – Transitions from cloud-based keep apps to local markdown files, enabling direct querying by local AI models while maintaining offline accessibility.
• Note Categorization – Divides information into secure work records and insecure personal logs, ensuring clean data pipelines for future retrieval and analysis.

03:50:00 Browser Extension Management & Security Isolation

Separates web browsing activities from primary work processes to minimize attack surfaces. Running dedicated user profiles prevents plugin conflicts and credential leakage.

• Jailed User Accounts – Creates restricted system profiles that only launch the browser, isolating extensions from core workstation operations.
• Shared Folder Synchronization – Establishes a single directory path bridging work and browsing users, allowing seamless file transfers without cross-contamination.
• Extension Audit Process – Leverages Chrome debug commands to enumerate installed plugins, verifying functionality before deployment on target networks.

06:15:00 Training Optimization & Audio Processing

Accelerates mandatory compliance viewing through speed manipulation and automated transcription. Converting video content into searchable text enables rapid information retrieval.

• Global Speed Control – Increases playback rates up to sixteen times normal speed, drastically reducing time spent on repetitive corporate training modules.
• Whisper Diarization Pipeline – Downloads video tracks, separates speaker voices, and generates timestamped transcripts for quick reference during assessments.
• Download Management – Employs multi-threaded swarm downloaders and classic turbo managers to handle bulk media retrieval without interrupting active workflows.

10:40:00 Virtualization & Network Tunneling Protocols

Establishes isolated testing environments using Windows virtual machines while managing connectivity constraints. Proper session handling prevents unexpected disconnections during remote engagements.

• Enhanced Session Mode – A Hyper-V feature providing higher resolution and shared clipboard functionality; disabling it is required before initiating certain VPN clients to avoid routing conflicts.
• Split Tunneling Mechanics – Routes specific traffic through the virtual network while keeping local resources accessible, preventing complete internet loss during connection tests.
• Certificate Verification – Identifies self-signed SSL mismatches early in the process, documenting them as preliminary findings before proceeding with authentication steps.

15:30:00 Macro Automation & Input Remapping

Remaps frequently used keyboard shortcuts to reduce physical strain and accelerate command execution. Running scripts with elevated privileges ensures reliable input registration across virtual environments.

• Caps Lock Repurposing – Converts the caps lock key into a primary modifier, assigning copy/paste functions to adjacent letters for faster workflow navigation.
• Physical Typing Macros – Simulates keystrokes with deliberate delays, allowing seamless data entry into restricted VM consoles that block standard clipboard operations.
• Administrator Execution Requirement – Highlights that macro scripts must run with elevated privileges to successfully inject inputs across different desktop sessions.

20:15:00 Portable Development Environments & Python Management

Deploys lightweight scripting frameworks that dynamically provision necessary tools without modifying host configurations. Verifying package contents prevents dependency conflicts during testing.

• Jamboree Framework – A PowerShell-driven utility that downloads and configures development stacks on demand, resetting environment variables to maintain system cleanliness.
• NuGet Package Filtering – Queries Microsoft's repository API to retrieve specific Python versions, ensuring compatibility with legacy tunneling scripts.
• Binary Verification Process – Checks extracted archives for bundled pip.exe or pip3.exe executables, eliminating manual module installation steps during rapid deployments.

28:40:00 AI-Assisted Scripting & Debugging Workflows

Generates and refines PowerShell functions through iterative conversational prompts. Validating AI output against actual system behavior prevents silent configuration errors.

• Vibe Coding Approach – Relies on continuous feedback loops with language models to draft, minimize, and debug automation scripts in real-time.
• Parameter Standardization – Enforces strict formatting rules for PowerShell commands, avoiding hardcoded paths and ensuring cross-environment compatibility.
• Temporary Storage Management – Monitors extraction directories to prevent disk saturation, redirecting large package downloads away from constrained system partitions.

35:10:00 Terminal Emulation & Advanced Tunneling Strategies

Facilitates complex network routing through dedicated terminal applications. Configuring dynamic and static tunnels enables reliable reverse connections for remote assessments.

• MOBA Portable Configuration – Utilizes an INI-based tunnel manager that automatically maintains connections across changing IP addresses or Wi-Fi networks.
• Reverse Shell Routing – Establishes outbound channels back to the tester, then proxies all subsequent traffic through those connections for consistent monitoring.
• Proxy Chain Integration – Forces non-proxy-aware applications to route through Burp Suite or custom interceptors using Windows utility wrappers like Priboxy.

42:30:00 Final Connectivity Testing & Engagement Wrap-Up

Executes comprehensive port scans to verify target accessibility before documenting findings. Acknowledging workflow detours ensures realistic time management during active engagements.

• Nmap Verification – Runs full-port scans with verbose output to confirm host responsiveness and identify open services prior to credential testing.
• Connection Refusal Documentation – Captures screenshot evidence of failed routing attempts, providing clear proof of network restrictions for client reporting.
• Workflow Reflection – Recognizes that exploratory debugging adds value but requires time boundaries; balancing thoroughness with engagement scope maintains professional efficiency.

Provide feedback on this episode.

As a child, I was so fond of reading what I saw as beautiful. Even when above [lacking] the understanding.

The perception of beauty came mainly from the cover and the fonts used ( same today ) or from the place the book occupied on my father’s shelf.

One of them was Tuareg. More than once I took this classic from the home library, only to glance at the first page with no interest. It was only decades later — some years ago — that I saw the title again, and read with pleasure and no difficulties. It’s not a hard read, even to a 12 year-old, probably only the argument, the plot, didn’t catch me by then.

Anyway, I was attracted to the book, as if simply wanting to read it made me part of the classic, the intellectual.

I have less of this today. [I mean,] Not the inability to read a lot of books, but the urge to be a consumer of certain cultural product. Maybe due to not greeding to build a personal image, content without exposition.

Still, sometimes, wanting to consume a piece of media and being unable to.. I wanted a story of robots, the book I though was not available [(All Systems Red)], so I bought one of scientific fiction.

Somehow it attracted me when I flipped the few pages of Planet of Exile at the bookstore, an item that could lead me to a travel outside of this place, an escape. At home, I immediately saw I couldn’t grasp it. There were too many characters (too many = more than two in the first page, specially when another atmosphere is the location, a new physics to grasp). Too much information, can’t retain the minimum necessary in the way to the second, third… fifth page, by when more is being presented. This was always a disability of mine, focused on a little tiny thing, intensely, for some time, all lost in some weeks or months. I know it happens to me , because people are able to remember the names of the cars of the movie Cars. Not an incredible feat for them (maybe they haven’t paid attention, nor actively tried to remember, only did), and something strange to me.

It was decided to register this because when I saw myself unable to read the [Ursula] Le Guin’s book, I got nervous. I’ve already been forgetting more ultimately, take confused steps, and now I can’t center on a book?! What’s happening?

But then I took a breath: much can be going way too wrong from current circumstances outside and inside, but your limited comprehension of the world (and its art) is very part of you. This trace, or frailty, of your character, is not your brain rotting, it’s part of what you have been since always.

Thank you.

Provide feedback on this episode.

In his early days as a writer, Heinlein wrote his stories in the context of a shared universe that he called the Future History. These were mostly short stories at first, with the occasional novella. But they include some great stories.

The Future History, Part 2

There were a few key themes running through Heinlein’s body of work. One we have already remarked upon, individual freedom, which had to be protected from any source of power, including both government and private corporations. This was essentially a libertarian perspective, but unlike many of today’s libertarians he was equally averse to the corporate type of power as a threat. But he had a complex view of the world which has resisted some attempts to pigeonhole him. He started out as a socialist, and while he didn’t remain one, he never became a knee-jerk reactionary either. In fact, he clearly despised them just as much. One way of looking at his body of work is that he explored the ramifications of different social policies through his stories, but in most cases the needs of a good story came first in the early years. In his later works he often surrendered to the temptation to pontificate, which reduced the enjoyment of them somewhat for anyone who was not already in agreement with his opinions

The second major theme you see throughout all of his works is the idea of the competent individual. He admired anyone who could do a job well, and clearly did not care whether they were man or woman, nor black or white. Alexei Panshin writes, in Heinlein in Dimension:

“There is one unique and vivid human Heinlein character, but he is a composite of Joe-Jim Gregory, Harriman, Waldo, Lazarus Long, Mr. Kiku, and many others, rather than any one individual.  I call the composite the Heinlein Individual.  . . .  It is a single personality that appears in three different stages and is repeated in every Heinlein book in one form or another.

“The earliest stage is that of the competent but naïve youngster. . . .  The second stage is the competent man in full glory, the man who knows how things work. . . .  The last stage is the wise old man who not only knows how things work, but why they work, too.”

Harriman we have already encountered in The Man Who Sold The Moon, and the others appear later. The Heinlein Individual, as he is often referred to, appears in many of Heinlein’s stories.

A third major theme has to do with morality and religion. Heinlein grew up in what he considered the heart of the Bible Belt, in Missouri, and saw first-hand how the evangelical Christians operated, and despised what he saw. As someone who believed in individual freedom, he could never surrender to someone else’s idea of how he should live his life. He saw them as a danger to his ideal libertarian society, and this shows up very early in his work. He personified the good, upright, church-going folk as “Mrs. Grundy”, and while you might want to draw the drapes to keep her from knowing what you were doing, you should never let her dictate how you would live your life. Revolt in 2100 begins the exploration of this in detail.

There is a chart of the future history at Baen Books, and in it we see that the 1960s were what Heinlein called The Crazy Years. (Remember, he conceived this in the 1940s and 1950s.) But in 2012 the major thing occurred when Nehemiah Scudder, a backwoods preacher, managed to get elected as President. This would be the last election held under the U.S. Constitution as he established a religious dictatorship that lasted a couple of generations. IS this plausible? Heinlein wrote about this:

“As for … the idea that we could lose our freedom by succumbing to a wave of religious hysteria, I am sorry to say that I consider it possible. I hope that it is not probable. But there is a latent deep strain of religious fanaticism in this, our culture; it is rooted in our history and it has broken out many times in the past.

“It is with us now; there has been a sharp rise in strongly evangelical sects in this country in recent years, some of which hold beliefs theocratic in the extreme, anti-intellectual, anti-scientific, and anti-libertarian.“

His background in the Bible Belt is what informs a lot of his thinking. He goes on to describe how this might happen:

“Throw in a Depression for good measure, promise a material heaven here on earth, add a dash of anti-Semitism, anti-Catholicism, anti-Negroism, and a good large dose of anti-“furriners” in general and anti-intellectuals here at home, and the result might be something quite frightening — particularly when one recalls that our voting system is such that a minority distributed as pluralities in enough states can constitute a working majority in Washington.”

As the science fiction author David Brin points out, Heinlein accurately predicted much of what we are going through in the United States right now. There is an emerging dictatorship in the United States, promoted by right-wing religious groups. The “material heaven here on earth” is represented by the Prosperity Gospel, prominent in the Trump movement, and so on. Where the Prophet used a restored Ku Klux Klan as his muscle, we have The Proud Boys, and so on. It really does track very closely. Read David Brin’s article for more on this.

But nothing lasts forever. Empires rise and fall, governments change, and in this case a resistance movement arises. The revolt is depicted in the novella If This Goes On— (1940), and it is set in the year 2100, giving the title to the book. The main character is John Lyle, who is a young army officer assigned to the group protecting The Prophet in his capital of New Jerusalem. In the beginning he is thoroughly indoctrinated, but then begins to question his beliefs when he falls for one of The Prophet’s virgins, Sister Judith. He has an older companion in the military who is not only unshocked when John confides in him about his doubts, but offers to help him. It turns out this companion, Zeb Jones, is a member of the underground group called The Cabal that is working to overthrow the theocracy. In the end they are successful, and in the course of this John Lyle does a lot of growing up. In this we see another common characteristic of Heinlein stories: a young, naive boy meets up with an older and wiser man who helps him to grow.

In 2016 If This Goes On— won the Retro-Hugo Award for best novella of 1940. And in a personal note, I have T-shirt that says “Scudder for President 2012”. This baffles most people, but I enjoy the in-joke.

What is interesting in this book is that Heinlein doesn’t stop with a successful revolution. He then goes on in a second novella to describe the government that arose following the revolution, and this story is called Coventry (1940). The new government that arises after the revolution is called The Covenant, and it is an attempt to make sure that what happened with Scudder in 2012 could never happen again. It is a strongly libertarian government based on an agreement to be non-violent. In this society, scientists can cured criminal or violent tendencies, but any citizen convicted of such must agree to the treatment. The alternative to treatment is that they can be exiled to a place called Coventry. Coventry is outside of the Covenant society, and the Covenant society has nothing to do with them.

Our protagonist, David McKinnon, is convicted of assault, and chooses to go to Coventry instead of getting treatment. He imagines it is a peaceful anarchy, but is disabused of this notion when he is robbed of all of this possessions upon entry and thrown in jail. A fellow inmate, Fader Magee, helps him escape, and we learn he is an agent of the Covenant government. They learn that two of the factions in Coventry have joined forces, and found a way to break through the barrier that surrounds Coventry. They plan to attack and overthrow the Covenant government. David and Fader separately work to escape and get back to warn the Covenant government, which they do successfully. And by doing this, David has demonstrated that he is no longer a danger to the Covenant society and no longer subject to treatment.

This story won a Prometheus Hall of Fame Award, which is awarded by the Libertarian Futurist Society. And the Covenant society certainly has libertarian features. But this is not the Randian version of libertarianism, as exemplified by the fact that David is restored to the society because he demonstrated his concern for others. Heinlein always promoted individual freedom, but also the idea that people have a responsibility towards others.

Finally, Revolt in 2100 contains the short story Misfit, w2hich we have looked at previously.

The Past Through Tomorrow (1967) is a one volume collection of most of the Future history stories. I say most because just which stories belonged in this group could change from time to time. It also has the last version of the Chart of the Future History, and a few stories we have not yet mentioned (Methusaleh’s Children, and The Menace From Earth). And many of his other works contain back references to these events that imply that they might be set in the same alternate universe. Heinlein gets the last word on this:

“I have never been sure whether or not publishing that chart was a good idea or a bad mistake. Possibly it helped to sell some stories later—but certainly it caused me and still causes me to receive a lot nuisance mail from nitpickers. I have never felt bound by that chart; it was to serve me, not the other way around. If I found myself with a good story notion which fitted fairly well into the chart but not perfectly, I shed no tears—I went ahead and let the inconsistencies stand.

I want each story to be internally consistent . . . but I won’t let myself be painted into a corner through trying to fit that chart perfectly. I may start another “Future History” story tomorrow . . . and find that to make it a good yarn I must violate some item on that chart. I’ll give the nitpickers something to pick, for I will not hurt a good yarn for the sake of “logic”—logic is not involved, as that chart is fiction, not Holy Writ.”

Links:

• https://www.amazon.com/Heinlein-Dimension-Critical-Alexei-Panshin/dp/0911682015
• https://web.archive.org/web/20151105170345/http://www.baenebooks.com/chapters/1439133417/1439133417___1.htm
• https://david-brin.medium.com/heinleins-future-history-coming-true-before-our-eyes-10356a95556a
• https://en.wikipedia.org/wiki/%22If_This_Goes_On%E2%80%94%22
• https://en.wikipedia.org/wiki/Coventry_(short_story)
• https://en.wikipedia.org/wiki/Prometheus_Award#Hall_of_Fame_Award_inductees
• https://en.wikipedia.org/wiki/The_Past_Through_Tomorrow
• https://www.palain.com/science-fiction/the-golden-age/robert-a-heinlein/the-future-history-part-2/

Provide feedback on this episode.

Kevie, Paul and Dave are back once again with the HPR Beer Garden, this time focusing on the relatively new style of beer called Double IPA. Kevie samples Man Overboard by Cromarty Brewing , Paul tastes Rumble by Beavertown and Dave opts for Ultra Phase by North Brewing

Connect with the guys on Untappd:

• Dave
• Paul
• Kevie

The intro sounds for the show are used from:

• https://freesound.org/people/mixtus/sounds/329806/
• https://freesound.org/people/j1987/sounds/123003/
• https://freesound.org/people/greatsoundstube/sounds/628437/

The next 3 beer styles to be reviewed:

• Belgian Blonde
• DDH IPA
• Amber Ale

Provide feedback on this episode.

01 Introduction

In this episode I will describe how to calculate elapsed time in bash or other shell scripts.

While this may sound like a very simple and basic thing to do, there is a slightly more complex aspect to it if you wish to calculate elapsed time to a higher resolution than one second.

02

There are many reasons for calculating elapsed time in a shell script.

For example you may wish to simply report how long an operation took to run.

Another reason may be that you are trying to speed up a script and need to calculate benchmark data to see how different alternative methods perform.

03

What may seem like a simple task gets a bit more complicated if you want to do it for multiple different operating systems even if they are all unix related, as we shall see.

--------------------

04 Operating Systems Tested

For the purposes of this episode, I ran tests on the current version of the following operating systems.

Alma

Alpine

Debian

FreeBSD

OpenBSD

RaspberryPi

OpenSuse

Ubuntu 2604

Alma is a close copy of Red Hat that we can take as representing Red Hat style distros.

--------------------

05 Simple Low Resolution Timing

I will start with the simple and obvious method before describing the less obvious ones.

This uses the date command to get the current time in seconds since the unix epoch.

This is simply

date '+%s'

06

Save this to a variable using whatever method you prefer.

For example.

starttime=$(date '+%s')

07

Next, do whatever operations it is you wish to time.

Use the date command to get the current time again.

endtime=$(date '+%s')

08

Now simply subtract the start time from the end time using shell arithmetic.

This should be very obvious and basic.

--------------------

09 Higher Resolution Timing

However, suppose we wish to measure time to greater than one second of precision.

We need to do two things.

The first is to obtain the current time at a higher degree of precision.

The second is to conduct the calculations to a higher degree of precision.

10

Unfortunately, the standard time precision for POSIX shells seems to be 1 second.

Some shells offer a higher precision, but others do not.

Furthermore, standard shell arithmatic uses integer, which limits calculations to 1 second of precision.

--------------------

11 Bash High Resolution Shell Variable

Fortunately, bash is one that does offer a high precision date.

If you are using bash 5.0 or newer, there is a shell variable called EPOCHREALTIME which offers time since the the unix epoch (that is, since the first of January 1970, at 00:00:00 UTC) in seconds to 6 decimals of precision.

12

Example

echo $EPOCHREALTIME

1779634800.184926

13

This is related to the similar bash variable known as EPOCHSECONDS which gives the number of seconds since the unix epoch.

14

Example

echo $EPOCHSECONDS

1779634800

15

So if you are using bash, measuring time is very simple.

--------------------

16 But is it Really Bash?

Is your script however actually using bash?

Debian and derivatives actually have two shells.

The first, the interactive shell is bash.

The second, the non-interactive shell is dash, which stands for "debian almquist shell".

17

If you open a terminal, you get bash.

If your script starts with a "bin/bash" shebang line, you get bash.

However, if your script starts with a "bin/sh" shebang line, you get dash.

Some people find themselves getting caught out by this one when they try something out in a terminal but find that it doesn't work in their script which started with "bin/sh".

18

Many other, but not all, Linux distros use bash for both the interactive and non-interactive shells, so "bin/sh" and "bin/bash" work the same with those ones.

So if you intend to use bash, make sure your script calls for bash in the first line.

--------------------

19 The SHELL Variable

So how can a script tell what shell it is running under?

There is a shell variable called "SHELL" which will tell you the name of the shell.

Well, sort of.

20

On Debian and derivatives "SHELL" will say "bash" regardless of whether the actual shell is bash or dash.

On some other operating systems "SHELL" will simply say "sh" even if it is something else entirely.

So we need to do some additional levels of checking to see what we have.

21

To start with though, here's what each of the test distros reports for SHELL.

Alma : bash

Alpine : sh

Debian : bash

FreeBSD : sh

OpenBSD : ksh

Raspberry-Pi : bash

Suse : bash

ubuntu2604 : bash

--------------------

22 Bash Versus Dash

First, let's try to see which ones are bash and which ones are dash.

The first thing we can check is for the shell variable BASH_VERSION.

23

Example

echo $BASH_VERSION

If the shell is bash, then it will report a version string.

If the shell is not bash, then it will return an empty value.

24

Using this test, we can see that Alma and Opensuse are indeed using bash.

We however need to check Debian, Raspberry Pi, and Ubuntu when running in an "sh" script.

To check this we can use the "which" command to see what "sh" actually is.

25

Example

echo $( ls -l $(which sh ) | rev | cut -d" " -f1 | cut -d/ -f1 | rev )

26

"which sh" shows us the path to "sh"

However, that is a link so we need to use

"ls -l" to find the actual executable.

"rev" reverses the string.

27

"cut" takes the first element separated by spaces.

The second

"cut" takes the first element separated by the "/" characters.

The final "rev" takes that string and reverses it again to get it in the correct order.

28

In the case of Debian, Raspberry Pi, and Ubuntu it tells us that this is "dash".

--------------------

29 Openbsd

Openbsd reports its shell as "ksh", which stands for Korn Shell.

It is indeed Korn Shell, so we simply leave that one as is.

--------------------

30 Alpine and Freebsd

Next we have Alpine Linux and Freebsd, which both report as "sh".

In the case Freebsd there doesn't appear to be any further we can go that I am aware of.

It's simple "sh".

It is a basic POSIX shell which seems to be similar to the original unix shell, the Bourne Shell.

Older versions of Freebsd used a different shell known as tsch (the C shell), but I haven't tested that so I will ignore that here.

31

With Alpine Linux however, we can get the actual shell using the same method that we used for Debian Linux.

This reports as being "busybox".

32

Busybox is a limited shell intended for use in embedded systems.

Alpine was originally an embedded distro, but some people started using it for containers.

Alpine is Linux, but it is not GNU/Linux, and there are a number of areas which can trip you up if you are not aware of them.

So, be extra careful if you are using it for anything, and test everything.

--------------------

33 Summary of Actual Shells

Here is our revised list with the actual shell used when asking for "sh", so far as we can determine.

Alma : bash

Alpine : busybox

Debian : dash

FreeBSD : sh

OpenBSD : ksh

Raspberry-Pi : dash

Suse : bash

ubuntu2604 : dash

34

There are other shells, but none of them are the default shell for any of the distros on our list, so I haven't tested them.

--------------------

35 Solutions for Measuring Time

Now we need to find solutions for bash, dash, ksh, sh, and busybox.

--------------------

36 Bash

For bash, we can simply use EPOCHREALTIME, as mentioned above.

--------------------

37 Dash

For dash, we can use the date command.

This is a very conventional method, and is probably the first answer that anyone would give for this situation.

However, while it will work in most cases, it will not work in all cases, so it is not a universal solution.

38

To use date we simply call it with the correct format string.

This uses %s to get seconds since the epoch, and %N to get nanoseconds of the current second.

If you put a decimal separator between the two it will appear in the output.

You can use the correct decimal separator for your locale, but I won't go into that here.

Instead I will just assume a period or dot.

39

Example

date '+%s.%N'

1779634800.358916385

--------------------

40 Problems with Date on Alpine and Openbsd

Date will work for bash, dash, and sh on Freebsd.

However it will not work for ksh on Openbsd, or for busybox on Alpine.

41

With busybox on Alpine, it simply ignores the %N format specifier and prints out the epoch in seconds only followed by the decimal separator.

=

Example

date '+%s.%N'

1779634800.

42

With ksh on Openbsd it prints the epoch in seconds followed by the decimal separator and then the %N as a literal N.

date '+%s.%N'

1779634800.N

43

Fortunately we have alternatives for these two cases.

--------------------

44 Openbsd

Openbsd has the "ts" or timestamp utility installed by default.

ts prints a time stamp in front of every line it receives from standard input.

I won't go into details on all aspects of ts here, I'll leave that to someone else.

Instead I will focus on how to use it for our specific purposes here.

45

We need to provide a format specifier to ts, which in this case is "%.s"

We also need to provide something for standard input, or otherwise ts will simply sit there and wait for input.

So what we need to do is to echo nothing through a pipe to ts while also giving ts the proper format specifier.

46

Example

echo | ts "%.s"

47

This will output the epoch time in seconds to six decimals of precision.

ts is installed in Openbsd and Freebsd by default and can be used in either.

It can also be installed in many other distros.

--------------------

48 Busybox on Alpine

None of the methods discussed so far will work for busybox on Alpine though.

However there is a way, but it's a bit non obvious and somewhat hacky.

49

Busybox includes a command called "adjtimex".

This is normally used to adjust the time hardware.

However if it is run without arguments, it will report the current settings.

50

These include the current epoch time in seconds , and in another field the time in microseconds.

These are reported as key value pairs.

So what we need to do is to do the following

51

Run adjtimex

Capture the output.

Grep for "time.tv_sec"

Grep for "time.tv_usec"

Use cut to extract the time value in each case.

Use tr to get rid of excess spaces in each case.

Combine the two in a string with a decimal separator between them.

52

This takes a total of 4 lines of shell script.

I will just describe them breifly here, see the show notes for details.

53

First we want to capture the output of adjtimex in a single operation.

Run adjtimex and pipe the output through grep to capture lines containing "time.tv_"

and save this to a variable.

# Extract the current high resolution time from adjtimex.

# We want two key value pairs, identified by time.tv_sec and time.tv_usec.

tvals=$( adjtimex | grep "time.tv_" )

54

Next echo the contents of this variable and pipe it through grep, cut, and tr to get first the seconds and then the microseconds while also removing excess spaces.

Save these to two separate variables.

"time.tv_sec" is the time in seconds since the epoch.

"time.tv_usec" is the number of microseconds in the current second.

# Get the time since the unix epoch in seconds and micro-seconds.

timesec=$( echo "$tvals" | grep "time.tv_sec" | cut -d: -f2 | tr -d " " )

timeusec=$( echo "$tvals" | grep "time.tv_usec" | cut -d: -f2 | tr -d " " )

55

Adjtimex does not zero pad the microsecond time value to provide leading zeros, so we need to take care of this using printf before we can append it to the seconds value. We didn't need to do this with date where the %N format character does this automatically.

In this instance, the printf format string is '%06d'

padusec=$( printf '%06d' $timeusec )

Now, combine these into a single number with a decimal separator by using simple string concatenation.

# Combine them into a single number.

timehires="$timesec"".""$padusec"

--------------------

56 Summary of Methods

Let's summarize where we are so far in terms of methods we can use to get the current time as a high resolution number.

Alma : use EPOCHREALTIME or date

Debian (bash) : use EPOCHREALTIME or date

Raspberry-Pi (bash) : use EPOCHREALTIME or date

ubuntu2604 (bash) : use EPOCHREALTIME or date

Suse : use EPOCHREALTIME or date

Debian (dash) : use date

Raspberry-Pi (dash) : use date

ubuntu2604 (dash) : use date

Alpine : use adjtimex and parse the output

FreeBSD : use date or ts

OpenBSD : use ts

--------------------

57 Other alternatives

There are a few alternatives that we haven't discussed yet.

58 Bash with Dash

In the case of Debian, Raspberry Pi, and Ubuntu running dash, since bash is available it is possible to write a separate bash script which simply echos EPOCHREALTIME and then call it from the dash script and capture the output.

While this would work, there's probably not a lot of point to it.

If you can rely on bash being there, then just change the first line of the script and make it a bash script.

59 Adding Packages to Alpine

The ts or timestamp utility is a common unix utility that can be installed if it is not present by default.

This does produce high resolution timestamps on Alpine.

On Alpine Linux this comes as part of the "moreutils" package.

To add the package, use the following

sudo apk add moreutils

echo | ts "%.s"

1779634800.959948

60

You can also add the GNU coreutils, which will provide a high resolution date command which works like in the other examples.

To add the package use the following

sudo apk add coreutils

date '+%s.%N'

1779634800.212897332

61

If you can install more packages into your Alpine system, either of the above two is probably going to be preferable to parsing the output of adjtimex.

62 Custom Timestamp Programs

You can also write a very short program in python, perl, tcl, or some other language and have it output the current epoch time.

I won't discuss that here though.

--------------------

63 Calculating Time Differences

Shell arithmetic is integer only.

If we wish to use high resolution timing data, we need to do something so we don't lose the precision we have worked so hard to get.

There are several possible solutions.

64 Change the Time Base

One method is to change the time base from seconds to milli, micro, or nanoseconds.

This can be done by simply multiplying the time values by the appropriate amount (e.g. 1000, 1,000,000, etc.) before subtracting them.

This allows for integer arithmetic on high resolution values without losing precision.

65 Use the Shell bc Arbitrary Precision Calculator

The bc command line calculator will perform calculations using real numbers and is easy to use in scripts.

It is present by default in most distros.

echo "scale=9; $endtime - $starttime" | bc

where endtime and starttime are variables containing time values.

66

However, for some inexplicable reason, neither Debian nor Opensuse install it by default.

It is present in Ubuntu and Raspberry Pi which are Debian derivatives, and it can be added to distros which lack it.

67 Use awk

awk can also perform calculations using real numbers and it is present in nearly all distros including in all of the ones we tested here.

echo "$endtime $starttime" | awk '{printf "%.6f\n", $1 - $2}'

--------------------

68 Benchmarks

And of course no comparative evaluation would be complete without benchmarks where we see how each method compares to another in terms of speed.

In the benchmark test I ran each method in a loop through multiple iterations, measured the elapsed time, subtracted out the time for an empty loop, and then compared it to alternate methods.

For anything other than EPOCHREALTIME, the empty loop time is negligible and has no real effect on the results.

69

Rather interestingly I came across a bug which caused date to run very slowly if called immediately after using EPOCHREALTIME in bash.

The effect of the bug was to make the date benchmark test roughly 24 times slower.

This has been fixed in newer releases, but if you are using an older distro release then beware of this bug.

I was able to get around it either putting a sleep delay between benchmarking EPOCHREALTIME and benchmarking date, or by simply testing date before testing EPOCHREALTIME.

70

To be able to conduct additional tests I installed ts in Ubuntu and Alpine, and the GNU version of date in Alpine.

71 EPOCHREALTIME Versus date in Ubuntu 2604 bash

The EPOCHREALTIME method is 3103 times faster than date.

However, when the same test is run on Ubuntu 2404 when the date test is run before the EPOCHREALTIME test, EPOCHREALTIME is 1240 faster than date.

Other Linux distros show performance to Ubuntu 2404.

It appears that a side effect of fixing whatever the bug is has the effect of slowing down date.

However, this is probably not a significant issue in normal circumstances.

72 date versus ts in Ubuntu 2604 bash

The date method is 3.7 times faster than ts

73 date versus ts in Ubuntu 2604 dash

The date method is 4.9 times faster than ts

74 date versus ts in Freebsd sh

The date method is 2.5 times faster than ts

75 date versus adjtimex in Alpine Busybox

The date method is 6.0 times faster than adjtimex

76 date versus ts in Alpine Busybox

The date method is 20.0 times faster than ts

77 bc versus awk in Ubuntu 2604

I compared calculating the difference between two numbers when using bc versus awk.

The difference is negligible, with bc being only 7% faster than awk.

78 Conclusion for Benchmarks

Based on these results, if you need to measure elapsed time to high resolution and care about runing the command with as little overhead as possible, then the order of preference should be the following.

79

If you are using a newer version of bash, then use EPOCHREALTIME.

If that is not available, then use date, provided it allows for high resolution times.

If the above two cannot be used, then use ts.

If you are using Busybox and cannot install either GNU date or ts, then use adjtimex.

Date is the closest in terms of being the universal portable solution, but it does not work in all cases.

80

I have not compared different platforms to each other in terms of performance, as that would be a much more involved problem that is outside the scope of this episode.

However, different operating systems implement different commands in different ways.

81

For example, on Openbsd and Freebsd, ts appears to be an ELF binary. That is, it is executable machine code, possibly written in C.

On Ubuntu however, ts appears to be a perl script.

As a result of this, the advantage that date has over ts is much less in Freebsd than it is with Ubuntu (and likely other Linux distros) as on Freebsd it doesn't need to load a perl interpreter to run ts.

--------------------

82 Overall Conclusion

You no doubt thought that measuring elapsed time was going to be so simple, and how could someone get an entire podcast out of such a simple subject?

And yet here we are half an hour later with just a basic overview of the subject.

83

I hope you found this interesting and informative.

Please let us know in the comments if you think that I have done anything incorrectly, or if you have another way of doing things.

I hope to see you all again in another future episode of HPR.

--------------------

Provide feedback on this episode.

Learn more about Nicks work at calyxinstitute.org and phreeli.com.

Check out Cindy’s book Privacy’s Defender: My Thirty-Year Fight Against Digital Surveillance (https://amzn.to/4gXuK2J).

Sponsors

Support for this show comes from ThreatLocker®. ThreatLocker® is a Zero Trust Endpoint Protection Platform that strengthens your infrastructure from the ground up. With ThreatLocker® Allowlisting and Ringfencing™, you gain a more secure approach to blocking exploits of known and unknown vulnerabilities. ThreatLocker® provides Zero Trust control at the kernel level that enables you to allow everything you need and block everything else, including ransomware! Learn more at www.threatlocker.com.

Support for this episode comes from NetSuite. NetSuite gives you visibility and control of your financials, planning, budgeting, and of course - inventory - so you can manage risk, get reliable forecasts, and improve margins. NetSuite helps you identify rising costs, automate your manual business processes, and see where to save money. KNOW your numbers. KNOW your business. And get to KNOW how NetSuite can be the source of truth for your entire company. Visit www.netsuite.com/darknet to learn more.

This show is sponsored by Maze. Maze uses AI agents to triage and remediate cloud vulnerabilities by figuring out what’s actually exploitable, not just what’s theoretically risky. They remove the noise, prioritize vulns that matter, and manage remediation, so your team stops wasting time on meaningless vulns. Visit MazeHQ.com/darknet for more information.

View all active sponsors.

Sources

• https://www.democracynow.org/2010/8/11/gagged_for_6_years_nick_merrill
• https://globalfreedomofexpression.columbia.edu/cases/u-s-nicholas-merrill-v-loretta-e-lynch-14-cv-9763-vm/
• https://clearinghouse.net/case/12966/
• https://www.theguardian.com/law/2015/dec/06/fbi-national-security-letter-gag-order-nick-merrill
• https://www.aclu.org/documents/national-security-letters
• https://www.eff.org/cases/re-matter-2011-national-security-letter
• Cindy’s Book: Privacy’s Defender: My Thirty-Year Fight Against Digital Surveillance

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.

In UNIX Curio #8 ( HPR episode 4657 ), I talked about using standard utilities to compare files. Left unmentioned, however, was a method commonly used today—the hash function.

As I've stated in previous entries, while I am an engineer, I don't have a background in computer science, so my understanding of the mathematics is limited. But I can give a practical description of what a hash function does. It takes an input, performs a set of calculations on it, and produces an output. As hash functions are practically used, the input is a set of bytes, such as a file or another piece of data like a password. The output is a numerical value in a fixed range—most often, expressed as hexadecimal characters. Because this "hash value" can always be represented in a certain number of bytes, its length as printed is usually a constant number of characters, padded with leading zeros if necessary. This episode will not cover the use of hashes in programming, focusing instead on using them to validate data.

A hash function, or more specifically, a cryptographic hash function, has an additional property. It should be very difficult to predict what changes to the input would be required to produce a specific change in the output.

An older, related concept is called a "checksum". While these are designed to vary when the input data is damaged or digits are transposed, they do not necessarily have that last property mentioned for cryptographic hashes. You have probably already encountered a checksum, even if you didn't recognize it. On a 16-digit number assigned to a Mastercard or Visa ¹ credit or debit card, the first six digits identify the card issuer (such as a bank), the next nine digits are assigned to you by the issuer, and the last digit is a check digit. The check digit is calculated using the values of the previous 15 digits, and it is a simple way to avoid typos in entering a card number.

In another example, every Ethernet frame that your devices send or receive includes a checksum ² to help ensure that the contents weren't scrambled in transit. This is 32 bits long and is called a cyclical redundancy check, commonly referred to as a CRC. A CRC is also used in many other places—for example, the .zip file format includes one for each archive member, and this allows a program extracting files from the archive to identify if any were damaged.

Our UNIX Curio for today is another example, the cksum utility ³ . It generates a 32-bit CRC based on the Ethernet algorithm. It operates on either a named file or standard input and outputs the CRC value, the length of the input, and the pathname if a file was given as an argument. Unlike most modern hashing programs, the checksum is printed as a decimal integer and is not padded, so it can be anywhere from one to ten digits long. The length value is the number of bytes in the input (actually specified as the number of octets , as systems could potentially use a byte that isn't eight bits long), also expressed as a decimal integer.

There are two major ways that one could use cksum to check the validity of a file. First, if you are transferring a file from one UNIX-like system to another, you could run cksum against it on both systems and check that the CRC and length are the same. The utility can also be given multiple filenames as arguments, which would generate a list that can then be compared. The second way would be for someone publishing a file or set of files to also publish the CRC values, lengths, and names so that people downloading them could verify that they match. However, I don't think the practice of publishing lists like this really started until more recent hash functions like MD5 and SHA-1 came about so it is unlikely that anyone would publish CRC values instead.

The advantage of these tools should be pretty obvious in comparison to cmp , one of the utilities discussed in UNIX Curio #8. To verify a file using cmp , you need two files to compare—if you're trying to check a large file you downloaded, you would need to spend the time and bandwidth to download a second copy. And if they didn't match, you would have no idea which of the two, if either, was correct. By contrast, cksum is quicker to run, doesn't require downloading a massive amount of excess data, and if run against the original file, makes clear what the correct value is.

This utility is a follow-on from a program called sum , which operated very much the same. I had a bit of trouble tracking down the exact development history, but what seems clear is that two different variants ⁴ were popular: a BSD version and a System V version. Both output 16-bit checksums, but used different algorithms so they didn't give the same results. Also, the BSD version printed the length of the input data as the number of 1,024-byte blocks, while the System V version instead gave a count of 512-byte blocks. (Some sources claim that System V sum generates a 32-bit checksum ⁵ , which could possibly be true internal to the algorithm, but I have tested several independent implementations of the utility and all of them output a 16-bit value for both the System V and BSD algorithms.)

From what I can tell, the BSD version ^6,7 came first; it was in 3BSD but probably appeared even earlier. An identical copy of BSD's sum was included with UNIX/32V ^8,9 , which was AT&T's 1979 port of Seventh Edition UNIX to the VAX and became one of the ancestors of System III. The divergence seems to have started with System III, released in 1980; its version of the sum utility ^10,11 changed to a new default algorithm, though it could be made to use the BSD algorithm via the -r option. System V looks to have kept the same behavior as System III. It's not clear to me why this algorithm is universally called the "System V algorithm" rather than the "System III algorithm"; perhaps it is because System V saw much more widespread use.

Instead of trying to reconcile these differences, the POSIX committee decided to create a new utility with a unique name, use a separate algorithm entirely, and avoid the block-length dispute by printing the length in octets instead of blocks. I should point out that POSIX states that the CRC algorithm for cksum does not strictly meet the mathematical definition of a "checksum". I don't know enough to say exactly why it doesn't qualify or to say whether either of the sum algorithms do. However, in less-formal usage the term "checksum" has gathered the meaning of any value used to represent or validate a set of data, so I am fine with using it no matter the technical details of the algorithm.

When two different inputs produce the same checksum or hash value, this is called a "collision". Because the output value has a limited range, there are an infinite number of possible inputs that could produce a collision. From a practical standpoint the possibilities are more limited—the majority of these inputs are larger than the number of atoms in the universe, which can't fit on any machine. Unlike a cryptographic hash algorithm, the CRC is not specifically designed to resist an attacker crafting a malicious input that would cause a collision. However, it should be sufficient to detect accidental damage.

Programs implementing more modern cryptographic hash algorithms are superior to the checksum utilities in avoiding collisions (whether malicious or accidental), but there are still three advantages that the older programs have. First, a system running a historical operating system might not have the hash programs available, but is more likely to have cksum or sum already included. Second, the checksum values are much shorter than the hashes output by the newer programs, making them easier for a user to compare by looking at them. This advantage is not as great as it might appear at first, because a common way to check a hash these days is to save a list of hashes and filenames—the hash programs can use that and do the comparison themselves, sparing the user from having to validate it character by character. The third advantage is that cksum prints the input length in bytes. This greatly limits the number of inputs that could be maliciously crafted to create a collision.

I did a moderate amount of research on implementations of modern cryptographic hash algorithms and found that some, such as MD5, SHA-1, and SHA-2, do use the length of the input (often termed "message length" in the literature) as part of the material fed in to the algorithm, but none of the hashing utilities present this length to the user as part of its output. There are two possible reasons for this that seem evident to me. First, if one is hashing a password, you would certainly not want to give a clear indication of its length—that would give any attacker a massive head start on guessing the password. However, that doesn't explain why one would avoid printing the input length for a file that is made publicly available. Second, it is convenient in many contexts, such as database entries or in software (such as git ), for the hash to be a fixed length. Including an extra value that can be of variable length would complicate those use cases. However, the length value could simply be dropped and they would be no worse off than they are currently.

Historically on UNIX, password hashing was treated differently from checksumming files— the crypt() function ¹² was used for passwords while sum and later cksum were used to confirm a file's integrity. So even rather early on, these two use cases employed algorithms with different properties, but I haven't dived into the history deeply enough to know how intentional this was. My discussion in this episode focuses on the file use case, so understand that I'm largely avoiding the topic of password hashing. Digital signatures are yet another use case, one that I'm ignoring entirely.

Every few years, some security researcher declares a particular hash algorithm to be "broken" and that everyone should move over to a new one, which generally has a longer hash. While the larger hash space certainly reduces the opportunity for collisions, this disrupts workflows, such as publishing information about software releases by e-mail, which still tends to observe a 78-character limit on each line ¹³ , making it harder to include a list of hashes with filenames next to them. This is in addition to the work of modifying software and scripts to use the new algorithm and managing how to treat past data. It seems to me that publishing the input length along with the hash would make it far more difficult to craft a malicious input that matches both, but I haven't found discussion of that during my investigation. (See the Appendix for a possible implementation.) Perhaps someone listening can record a response episode for HPR explaining that.

References:

1. Payment card number https://en.wikipedia.org/wiki/Payment_card_number
2. Ethernet frame: Frame check sequence https://en.wikipedia.org/wiki/Ethernet_frame#Frame_check_sequence
3. Cksum specification https://pubs.opengroup.org/onlinepubs/009695399/utilities/cksum.html
4. GNU coreutils manual: sum https://www.gnu.org/software/coreutils/manual/html_node/sum-invocation.html
5. FreeBSD 15.0 sum manual page https://man.freebsd.org/cgi/man.cgi?query=sum&sektion=1&manpath=FreeBSD+15.0-RELEASE+and+Ports
6. 3BSD sum manual page https://www.tuhs.org/cgi-bin/utree.pl?file=3BSD/usr/man/man1/sum.1
7. 3BSD sum source https://www.tuhs.org/cgi-bin/utree.pl?file=3BSD/usr/src/cmd/sum.c
8. UNIX/32V sum manual page https://www.tuhs.org/cgi-bin/utree.pl?file=32V/usr/man/man1/sum.1
9. UNIX/32V sum source https://www.tuhs.org/cgi-bin/utree.pl?file=32V/usr/src/cmd/sum.c
10. System III sum manual page https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/man/man1/sum.1
11. System III sum source https://www.tuhs.org/cgi-bin/utree.pl?file=SysIII/usr/src/cmd/sum.c
12. Crypt specification https://pubs.opengroup.org/onlinepubs/009695399/functions/crypt.html
13. RFC 2822: Internet Message Format: Line Length Limits https://datatracker.ietf.org/doc/html/rfc2822#section-2.1.1
14. OpenSSH 10.1 released https://lwn.net/ml/all/dd12623ae86aa5eb@cvs.openbsd.org/

Appendix

The MD5 hash algorithm was (and still is) widely used, but many people characterize it as being "broken" and discourage its use. Let us imagine a variant of this, called MD5.L, where the normal MD5 hash is followed by a "." character and the input length expressed as a hexadecimal number.

Take, for example, the e-mail message announcing the release of OpenSSH 10.1 ¹⁴ . At the bottom, it includes an SHA-1 hash and an SHA-2 256-bit hash for the available gzipped tar files. That longer hash is encoded with Base64 because if it were given as a hexadecimal number, it would make the line longer than 78 bytes. The MD5.L hash of the file would be one character shorter than the SHA-1 hash, as shown below. (The extra length of the name makes them both consume the same number of characters. The hashes shown are for the "portable" version of OpenSSH.)

Some people claim SHA-1 is also broken, seeking to have people use newer and longer hash functions. For an attacker to compromise MD5.L in this example, they would not only have to create a valid tar file compressed with gzip containing a malicious payload having the right MD5 hash, that file would have to be exactly 1,972,831 bytes long (the decimal equivalent of 1e1a5f). While there are still many possible inputs that could be tried (256 ^1972831 , to be exact*), this is far fewer than the infinite possibilities for plain MD5, SHA-1, or SHA-2.

If for some reason it is super important to have a fixed hash length, let's imagine another variation called MD5+L. In this one, instead of L being the input length, it is the input length modulo one terabyte (2 ⁴⁰ bytes), which can be represented by 10 hexadecimal characters, left-padded with zeros. While this approach substantially increases the number of possible inputs an attacker could try, it is likely that an intended victim would notice that the file they downloaded is larger (or smaller) than expected by that much. The MD5+L hash is longer than a SHA-1 hash, but still shorter than a 256-bit SHA-2 hash.

SHA1 (openssh-10.1p1.tar.gz) = 7fd17b99d1beffb47cd380d64079e920bb0bd91f
SHA256 (openssh-10.1p1.tar.gz) = ufx6K4JXlGem8vQ+SoHI4d/aYU3bT5slWq/XAgu/B1g=
MD5.L (openssh-10.1p1.tar.gz) = 80dd9bb00a86519934710d05903fdf07.1e1a5f
MD5+L (openssh-10.1p1.tar.gz) = 80dd9bb00a86519934710d05903fdf07+00001e1a5f

Of course, if MD5 is considered to be too weak even with the inclusion of the length, one could produce a ".L" or "+L" version of any hash function. However, longer hashes will end up running into the 78-character limit.

*This is a number with 4.75 million digits that the bc utility on my laptop took almost 5 minutes to calculate.

Provide feedback on this episode.

New hosts

Welcome to our new hosts:

• Lennart Benschop

Last Month's Shows

Comments this month

Past shows

• hpr4181 (2024-08-12) "Downloading out of copyright movies" by Bob.
  
• Jan said: "subtitles" (2026-06-29 21:55:12)
  
• hpr4633 (2026-05-06) "Ham Radio Licence" by Lee.
  
• RJ said: "Very interesting as always" (2026-06-02 11:22:47)
  
• Lee said: "Aerials" (2026-06-02 16:44:45)
  
• hpr4644 (2026-05-21) "Response to comments on HPR4424: Newsboat..." by Archer72.
  
• Archer72 said: "Not quite a complete script" (2026-06-07 22:26:34)
  
• Whiskeyjack said: "Response to hpr4644" (2026-06-08 18:01:33)
  
• Archer72 said: "Response to WhiskeyJack" (2026-06-12 16:50:42)
  
• Whiskeyjack said: "Response to Archer72 on HPR4644" (2026-06-13 11:13:49)
  
• Archer72 said: "EyeD3" (2026-06-13 21:10:48)
  
• hpr4649 (2026-05-28) "What did I do at work today? Part 3 Section 2" by Lee.
  
• Ken Fallon said: "Love PHP" (2026-06-02 10:52:54)
  
• candycanearter07 said: "Re: Love PHP" (2026-06-02 17:24:01)
  
• hpr4650 (2026-05-29) "Playing Civilization V, Part 12" by Ahuka.
  
• Antoine said: "Puppetting and Happiness" (2026-05-29 14:21:14)
  
• Kevin O'Brien said: "That's the algorithm" (2026-06-01 21:13:58)
  
• Antoine said: "#2 Thanks!" (2026-06-04 23:25:42)
  

This month's shows

• hpr4651 (2026-06-01) "HPR Community News for May 2026" by HPR Volunteers.
  
• candycanearter07 said: "busy weeknd" (2026-06-02 02:38:55)
  
• hpr4653 (2026-06-03) "Starting the Habit of Reading" by Thaj Sara.
  
• Antoine said: "Reading always has been a life-saver for me, this alone makes this show Very Nice" (2026-06-04 23:37:27)
  
• Kevie said: "Nice episode" (2026-06-10 08:11:43)
  
• hpr4654 (2026-06-04) "What's in my component Box?" by MrX.
  
• Kevie said: "Geeks are ahead of their time" (2026-06-10 08:14:12)
  
• hpr4655 (2026-06-05) "Tips for Glasses" by norrist.
  
• Trey said: "Avoid hand soap " (2026-06-05 12:11:33)
  
• Jim DeVore said: "Geeks are lazy, too" (2026-06-21 00:10:45)
  
• Operat0r said: "Glasses" (2026-06-26 18:53:44)
  
• hpr4656 (2026-06-08) "My review of musicozy sleep/exercise bluetooth headband" by Swift110.
  
• candycanearter07 said: "wishlist!" (2026-06-08 13:12:08)
  
• Kevie said: "Interesting" (2026-06-10 08:15:59)
  
• hpr4657 (2026-06-09) "UNIX Curio #8 - Comparing Files" by Vance.
  
• xmanmonk said: "Great Show (again)" (2026-06-09 23:31:58)
  
• candycanearter07 said: "comparisons" (2026-06-11 16:57:57)
  
• Whiskeyjack said: "HPR4657 - use of comm" (2026-06-13 11:47:53)
  
• Vance said: "Appreciate the comments" (2026-06-15 03:16:05)
  
• Whiskeyjack said: "Reply to Vance on awk in HPR4657" (2026-06-15 17:02:57)
  
• hpr4659 (2026-06-11) "Command Line Fun - Recording a show" by Kevie.
  
• candycanearter07 said: "ffmpeg concatenation" (2026-06-15 11:58:49)
  
• Whiskeyjack said: "Reply to candycanearter07 on HPR4659" (2026-06-15 16:53:16)
  
• hpr4661 (2026-06-15) "Laptop Computer Woes, or How I Learned to Love My Tech Hoarding" by Claudio Miranda.
  
• candycanearter07 said: "uses :)" (2026-06-16 15:04:20)
  
• xmanmonk said: "Great show!" (2026-06-17 04:37:48)
  
• ClaudioM said: "Thanks! (In reply to candycanearter & xmanmonk)" (2026-06-17 14:02:17)
  
• hpr4666 (2026-06-22) "How I got into tech" by Lennart Benschop.
  
• Windigo said: "Nice to meet you" (2026-06-26 02:24:48)
  
• candycanearter07 said: "hello!" (2026-06-27 15:04:02)
  
• hpr4667 (2026-06-23) "UNIX Curio #9 - printf" by Vance.
  
• candycanearter07 said: "learned something new :)" (2026-06-23 13:23:06)
  
• Whiskeyjack said: "HPR4667 - printf" (2026-06-24 03:54:20)
  
• xmanmonk said: "Another great show!" (2026-06-24 21:15:22)
  

Mailing List discussions

Policy decisions surrounding HPR are taken by the community as a whole. This discussion takes place on the Mailing List which is open to all HPR listeners and contributors. The discussions are open and available on the HPR server under Mailman.

The threaded discussions this month can be found here:

Events Calendar

With the kind permission of LWN.net we are linking to The LWN.net Community Calendar.

Quoting the site:

This is the LWN.net community event calendar, where we track events of interest to people using and developing Linux and free software. Clicking on individual events will take you to the appropriate web page.

Provide feedback on this episode.

SUMMARY

Presenter discusses modular inflatable systems and challenges with lawn ornaments.

IDEAS

1. Modular inflatable systems for improved durability.
2. Use of zip ties and 3D-printed parts for connectivity.
3. Challenges with air flow and electrical connections.
4. Need for modular lighting systems.
5. Repair techniques for inflatable seams.
6. Importance of quality materials for longevity.
7. Combining air and electrical systems in one tube.
8. DIY solutions for extending inflatable life.
9. Issues with cheap lawn ornaments and their components.
10. Industrial blow fans for large inflatables.
11. Protecting wires from weather damage.
12. Replacing faulty motors in inflatables.
13. Balancing air pressure to prevent leaks.
14. Modular design for easy maintenance.
15. Cost-effective solutions for outdoor decorations.
16. Challenges with outdoor charging boxes.
17. Enhancing durability through adhesive treatments.
18. Need for seamless integration of components.
19. Longevity of inflatables under continuous use.
20. Practical approaches to inflatable system design.

RECOMMENDATIONS

1. Use modular designs for easier repairs.
2. Incorporate 3D-printed connectors for durability.
3. Protect electrical components from moisture.
4. Opt for industrial-grade fans for large inflatables.
5. Combine air and electrical systems in one tube.
6. Apply adhesive treatments to prevent seam splitting.
7. Replace faulty motors with compatible alternatives.
8. Balance air pressure to avoid overfilling.
9. Use high-quality materials for longer lifespan.
10. Implement DIY repair solutions for cost-effectiveness.
11. Design seamless connections for improved functionality.
12. Prioritize weather-resistant components.
13. Create modular lighting systems for flexibility.
14. Test inflatables under continuous use conditions.
15. Use spray adhesives to extend inflatable life.
16. Simplify assembly for user-friendly installation.
17. Explore cost-effective materials without compromising quality.
18. Address common failure points in inflatable systems.
19. Develop standardized repair techniques.
20. Focus on integrated design for longevity.

Provide feedback on this episode.

Lee recorded a show 4511 which aired on November 17, 2025. Since I had some observations of my own I decided to record a show in response. I have enjoyed some significant series of audiobooks, both in Science Fiction and in History, as they are both topics I greatly enjoy. And many of these are available online.

Links

• https://www.openculture.com/freeaudiobooks
• https://librivox.org
• https://archive.org/download/IsaacAsimovAudioBookCollection
• https://www.google.com/search?client=ubuntu-sn&channel=fs&q=robert+heinlein+audiobooks+youtube
• https://archive.org/details/arthur-c.-clarke-audiobooks-two
• https://librivox.org/author/808
• https://archive.org/details/WillDurant-TheStoryOfCivilizationVolume1BookmarkableM4bFile

Provide feedback on this episode.

Hello, this is your host, Archer72 for another episode of Hacker Public radio.

In this episode, I make my first contact off a local repeater in a small town in Kentucky.

What got me to try my hand at radio? It was when I started to capture ISS (International Space Station) Ham radio transmissions. From Kentucky, I have logged receiving from Texas and Oregon and as far as a brief transmission over Great Britain.

My next step was to have a first contact, which I botched at first when I said my call sign, and didn’t leave enough time for someone to respond.

I thought that a repeater was the best place to start, so found this one on RepeaterBook.com

The repeater was found in RepeaterBook at RepeaterBook : My local Cynthiana, Kentucky repeater

Youtube : Harrison County Amateur Radio Club

QRZ : Harrison County Amateur Radio Club

Facebook : Harrison County Amateur Radio Club

The callsign for this repeater is Kilo-Four-Kilo-Juliett-Quebec

It is currently On Air (and quite active near club time and the weekend)

The base frequency is 147.165 MHz with an offset of plus (+) 600 KHz and a tone to open the repeater, of 67.0 Hz

It is hosted by the Bluegrass Amateur Radio Society

The RepeaterBook entry was updated on 2025-10-27

K4KJQ
On-Air
Open
 147.16500 (+) MHz 67.0

Cynthiana • Harrison County • Kentucky

Updated 2025-10-27


Technical
Downlink    147.16500
Uplink  147.76500
Offset  +0.600
Uplink Tone 67.0
Downlink Tone   67.0
Antenna (AGL)   125 feet
Sponsor: Bluegrass Amateur Radio Society
Sponsor: BARS 
Local Time  04:30 (EDT UTC-04:00 DST)
America/New_York
Reviewed    2025-10-27

Bluegrass Amateur Radio Society

RepeaterBook Worldwide

RepeaterBook is a worldwide amateur radio repeater directory

So, with after all that being said I programmed a Baofeng BF-F8HP to this frequency manually. You can also use open source Chirp software, but I wanted to know my radio to program it on the fly.

This brings me to a recording with permission from Keith (KO4BWJ) who is located outside this little A notable feature of this the town of Cynthiana, Kentucky is the hometown of the artist Robert Kirkman for the Television series The Walking Dead, and home to the mural depicting the main characters of the show. This mural had some controversy over the rights to have it depicted, or so I thought. The actual story is story is that some groups did not want a Zombie theme representing their town.

Cynthiana, Kentucky: Walking Dead Mural

Without further delay, here is the promised recording.

…

I hope you enjoyed and learned from this little piece of an experience Ham vs a new (to the airwaves) Ham.

This has been Archer72 (Kilo-Delta-Niner-Victor-Mike-Whiskey)

73

Provide feedback on this episode.

The Technical Dutch Open Source Event (T-DOSE) is a free conference to promote the use and development of Open Source software. This event has is organised yearly since 2006 in the Brainport region, near Eindhoven, The Netherlands. During this event, Open Source projects, developers and visitors can exchange ideas and knowledge.

Ken gave a presentation, you can watch the recording https://www.youtube.com/watch?v=6gmX-Ap7knI&t=10912s

Provide feedback on this episode.

In this episode, Operator dives into his ongoing journey to migrate away from centralized cloud ecosystems specifically moving his daily workflow off Google Keep and onto Obsidian hosted locally on a Debian server. Operating purely over a secure VPN to minimize his external attack surface, he discusses the security considerations of managing personal data in local plain-text markdown files.

The episode features a deep dive into local AI infrastructure, sparked by technologist Daniel Miessler’s recent shift away from RAG (Retrieval-Augmented Generation) in favor of a simpler, localized file-system-as-context approach (using fast search tools like ripgrep). Operator shares his own mixed results experimenting with RAG noting great success with massive, structured car repair manuals, but incredibly poor fidelity when indexing conversational podcast transcripts.

To find the sweet spot, Operator is testing a dual approach : combining flat-file local search with a PostgreSQL vector database ( pgvector ). He also rants about the frustrating "hype cycle" of online tutorials that claim to teach "local" setups but secretly rely on expensive, cloud-hosted frontier models.

Finally, the host introduces his ambitious roadmap for "Protocol AI." Designed as a localized, read-only dashboard to help manage his ADHD and "time blindness," this system will scrape, aggregate, and summarize his cluttered digital life including multiple Gmail accounts, Yahoo spam, calendars, and a massive array of social media feeds (Signal, Discord, Mastodon, BlueSky). The long-term goal? Transitioning from a read-only local summarizer to a safe, "human-in-the-loop" execution assistant that keeps his data out of the hands of mega-corporations.

References

Obsidian is a proprietary personal knowledge base and note-taking application that operates on Markdown files. The software is free for personal and commercial use; only the offered cloud services, optional commercial licenses, and early access versions are paid. It is available as desktop versions for macOS, Windows and Linux as well as for mobile operating systems such as iOS and Android, but not as a web application.

Obsidian - From Wikipedia, the free encyclopedia

Retrieval-augmented generation (RAG) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources. With RAG, LLMs first refer to a specified set of documents, then respond to user queries. These documents supplement information from the LLM's pre-existing training data. This allows LLMs to use domain-specific and/or updated information that is not available in the training data. For example, this enables LLM-based chatbots to access internal company data or generate responses based on authoritative sources.

RAG (Retrieval-Augmented Generation)

Provide feedback on this episode.

In our sample game we look at playing as Austria and aiming for a Diplomatic Victory. And our focus is on puppeting City-States, but I misunderstood and instead of making a Diplomatic Victory easier, it makes it harder. I still managed to get my Diplomatic Victory, but a Science or Domination Victory would definitely have been easier in this scenario.

Playing Civilization V, Part 13

Playing for a Diplomatic Victory, Part 2

So, how do you solve a Happiness problem? For a full discussion, you can check out this section of the Civilization Wiki. In Civilization V you have these options:

• Trade for luxuries – Luxury resources are the best way to get a quick hit of Happiness, and trade is the best way to do this quickly. But to trade you have to have met the other civ, so my drive to explore pays off here. Every luxury resource I can obtain adds +4 Global Happiness to my Empire. Note that only the first unit of the resources adds happiness. So if you have multiple copies of a luxury resource you should keep one and trade the others if you can.
• Buildings – Buildings like the Circus (+2), Colosseum (+2), Zoo (+2), and Stadium (+4) provide local happiness in the city that builds them. That still gets added to your total happiness number, but local happiness can never exceed the size of the population. So if you have a city with 5 population, building a Circus and a Colosseum might make sense, but building a Circus and a Stadium would not since 2 of the Happiness would not count. Buildings take longer than Trade deals, so if you need a quick hit start with trading for Luxuries.
• Wonders – Chichen Itza (+4), Notre Dame (+10), Taj Mahal (+4), Eiffel Tower (+5), Neuschwanstein (+1 for every Castle you have), Prora (+1, plus +1 for every 2 Social Policies you have adopted), and CN Tower (+1 per city). Wonders are even harder to build, and you may not get them since you are competing with other players. But there is a National Wonder worth getting, the Circus Maximus (+5)
• Social Policies – Every policy tree has policies that can add to Happiness or decrease Unhappiness, which is just as good.
• Natural Wonders – Each Natural Wonder grants +1 Global Happiness to each empire that has discovered it. So again exploration is key.
• Mercantile City States grant +3 Global Happiness to each Empire that is at least Friends with it.

So you have a lot of options here, and they are listed in order of how you should pursue them. Trading for Luxury Resources can be done quickly and provide an immediate boost, so you should be actively trading all along. Buildings and Wonders should definitely be pursued, but here you will probably be balancing demands for other investments as well. Natural Wonders are nice, but you need to explore the whole map to max that out. I was able to trade one of my Whales to Patchacuti for one of his Ivory, which helped push my happiness up. Then with Askia I traded one Spice for one Copper. I also made a small gift to the City-State of Manila to make them my ally. Once 5 turns have elapsed, I can take them over via Diplomatic Marriage. And I have started on Colosseums in several of my cities to push up my Happiness further.

Meanwhile I continued my exporations, and became the first to discover all of the the other Empires in the game. This makes me the host for the World Council, which is good for one additional vote. I will keep exploring since there are probably some Natural Wonders out there worth finding yet. And indeed on the next turn I found Mount Fuji, pushing my Happiness to 24. Then I Puppeted Manilaa, and over a couple of turns it dropped to 18. But a Colosseum got built, and I found a Natural Wonder,so I am back to 21. I am not the leader in this game, and I think I can afford to Puppet another City-State. For now I am avoiding doing this to Mercantile City-States since they can add to my Happiness. But there’s lots of City-States in this game. I picked Kathmandu for my next target, but also dropped 250 on Ragusa to become Friends with them. You have to look to the future sometimes. I am currently pulling in 144 Gold per turn, so I can afford it.

At turn 267 my Happiness was up to 30, but as soon as I puppeted Kathmandu it dropped to 22. This is why you need to manage your Happiness carefully. This pushed me another 100 points up on the Leader board, so I am in good shape. But with 22 Happiness, I can probably afford to start on puppeting Ragusa. At turn 272 I have 5766 Gold, and I am earninig 163 Gold per turn. My Happiness is at 27, so I am in good shape. I puppet Ragusa, and my Happiness falls to 18, but my status on the leader board goes up another 100 points, putting me 200 ahead of my nearest competitor. But a few turns later and my Happiness is back up to 24, so I am starting the process to puppet Bucharest. At turn 280 I have 6584 Gold, I am earning 179 per turn, and my Happiness is 26. Then I puppet Bucharest, and my Happiness drops to 16. But I am nearly 300 ahead of my nearest competitor on the leader board, so not bad.

Also, we are getting near to the first World Council, where I am the host. I have put a proposal on the agenda to enact a World’s Fair. The number 2 player has put a proposal on the ban Whales as a luxury resource, and I have a lot of whale resources. I suspect the World’s Fair will pass without my votes since many other players supported it, so I will put my votes into denying the Whale ban. We’ll see how it goes. I have the most votes, but if everyone else gangs up on me they can get it passed.

And as I expected, the World’s Fair passed without my votes, and the Whale ban was defeated. I tabled a proposal to enact Arts Funding for the next Council. It looks to be very popular. Right now I starting to snowball. On turn 287 I now have 6476 Gold, and I am earning 231 each turn. I have puppeted 6 City-States, and my Happiness is at 31. I can now start to puppet another. In the World Council I have 4 votes, and every other player has 1. My main strategy right now is to keep on friendly terms with the other players since everything is going well. But I am also starting to build factories to improve my Production capacity, which in turn wil let me update my Military. I picked up a Great Scientist, and used him to construct an Academy to boost my Science output. And every city next to a mountain has gotten an Observatory for added science . And then I constructed the Leaning Tower of Pisa and got one free Great Person of my choice, and I chose a Great Scientist and constructed another academy. And I hit a Golden Age good for 10 turns.

My Golden Age ended, but on the next turn I finished the Taj Mahal which gave me another 10 turn Golden Age, in addition to =4 Global Happiness. And I used some of my big cash balance to upgrade my military units, because in this game the best way to keep the peace is to have a strong military. Then on the next turn I was able to choose an ideology, so I picked Freedom, and for my two policies I chose Universal Healthcare and Capitalism, each of which added Happiness. I have earned a couple of Great Artists, and I plan to use them to create more Golden Ages since accumulating Culture is not part of my strategy. On turn 309 I puppeted Bratislava and started on Wittenberg. And now that I have expxlored the entire map I don’t need my Caravels any longer, so I deleted them. then on turn 314 my closest competitor, Japan, denounced me, and was in turn denounced by Brazil. I have been building up my military, so I am not worried. Basically, Japan has realized I am opening up a bigger lead, and that is why they denounced me. I’d like to win without going to war, but if he attacks me I can handle it. Meanwhile, I am also building up my Science to open up a lead there.

On the next turn I built the Red Fort, which improves my defensive strength. Brazil approached me to join in a war on Japan, but I declined. I am not seeking a military victory in this game, so I don’t want to be distracted. However I just puppeted two more cities, and looking at the map I now have 3 cities on the border of Japan. So maybe there will be a war in my future. I got another policy to choose, and picked Universal Suffrage, which increases the length of Golden Ages, since I plan a few. And I am starting to build more military units, starting with riflemen which are good for defending cities. If I am given time I will ship them to my puppeted cities near Japan. But if he attackes me before then, they are still worth having.

Links

• https://civilization.fandom.com/wiki/Happiness_(Civ5)
• https://www.palain.com/gaming/civilization-v/playing-civilization-v-part-13/

Provide feedback on this episode.

In the 14th episode of the HPR Beer Garden, Dave and Kevie are joined by regular guest Paul to sample super strong lager. Known in the media as 'Tramp Juice', the guys look at whether this infamous style is worth a try and how it rates as a beer when you look past the stigma. Dave samples Skol Super, Kevie tries Kestrel Super and Paul opts for Petra Mocna from Poland.

Upcoming beers:

Double IPA

Belgian Blonde

Double Dry Hopped (DDH) IPA

Connect with the guys on Untappd :

• Dave
• Paul
• Kevie

The intro sounds for the show are used from:

• https://freesound.org/people/mixtus/sounds/329806/
• https://freesound.org/people/j1987/sounds/123003/
• https://freesound.org/people/greatsoundstube/sounds/628437/

Provide feedback on this episode.

--------------------

01 Introduction

This is the second follow up to my 8 part series on nuclear power. In this episode I will attempt to answer a question posed by brian in ohio in a comment on HPR4583. In that comment he said:

02

--------------------

Loving this series. Maybe Whiskey Jack could give some cost comparisons between large and small reactors. He could also give us a realistic look at nuclear plant safety/accidents compared to conventional power production. Looking forward to the episode on FORTH generation reactors ;-)

--------------------

03

End of quote.

The first question I answered in my previous follow up, which was HPR4628. In this episode I will attempt to answer the second question, which was about the safety of nuclear power compared to other sources of electrical power generation.

One of the HPR janitors encouraged me to make this episode, so I think we can thank him for getting another HPR episode made.

04 Defining the Scope

First, let's define the scope of the question.

This will cover electrical power generation only.

Within that scope I will consider only the following sources of energy.

05

Coal

Oil

Natural Gas

Hydroelectric

Nuclear

Wind

Solar

I won't cover geothermal, wave, or tidal power as these are only used in very small amounts and so there simply isn't enough literature on them to base a discussion on .

06 Foreshadow Conclusion

I should mention right away that I cannot provide absolute answers to this question in the form of a nice, neat ranking table based on numbers from peer reviewed scientific sources.

The reasons for this will become apparent, but to put it briefly, the data on which to base such a ranking simply doesn't exist.

I will however provide context within which people can think about the issue.

Wherever possible, I will provide links to the references that I used in the show notes so you can read further on this yourself.

--------------------

07 Energy Catastrophism versus Energy Uniformitarianism

First though I need to go off on a slight geological detour in order to explain an important analogy that I will use.

08

In the 19th century there was a great debate among geologists over what is known as catastrophism versus uniformitarianism.

In seeking to explain the origins of the earth and of the landscape that we see around us, there were two points of view.

09

One was "catastrophism".

This is the belief that the mountains, valleys, and plains that we see around us were formed as a result of great catastrophes which occurred relatively recently in earth's history.

This explanation was necessary in order to fit geological features into an earth that was believed to be only a few thousands of years old.

This view was heavily influenced by religious belief.

In this view Noah's flood was the great catastrophe and the fossils of dinosaurs were the remains of animals who had not been saved on the ark and so had died in the flood.

10

The other point of view was uniformitarianism.

This was the hypothesis that the landscape we see around us can be explained by the very slow accumulation of very small changes over very long periods of time.

For this to be true however, the earth had to be far older than the few thousand years that a literal reading of the bible would suggest.

The earth in fact had to be many, many, millions of years old.

11

Eventually, the uniformitarian view won out and people understood that while some catastrophes can take place, the shape of the landscape is overwhelmingly due to small changes over very long periods of time.

12 How is this Relevant to this Episode You Ask?

How this is relevant is that I will use this analogy to explain how we need to think about energy and safety.

Very small numbers of deaths and injuries multiplied over many occurrences can add up to big numbers, comparable in scale or possibly even larger than a single catastrophe or even several of them.

13

I don't know if anyone else has used this analogy before, I have just thought of this when writing the script for this podcast.

None the less, I think it is a very useful way of helping to understand the issues.

14

As an example of this, think about the well known case of the safety of flying versus the safety of travelling in your car.

Air crashes are catastrophes that make the headlines.

Automobile crashes are seldom more than local news at best.

You have probably heard many times the claim that if you making a trip somewhere, you are safer to fly than to drive yourself in your car.

15 Example - Hydro versus Solar

I will now present an example of this.

Hydro electric power has some notable large scale catastrophes associated with it.

Roof top solar power does not have any notable catastrophes that I am aware of.

However, which is safer?

16 Hydro Catastrophes

Here are three examples of hydro electric catastrophes in just one country, Italy.

The Vajont Dam which collapsed in1963

An estimated 1,917 to 2,500 people died.

The Sella Zerbino dam which collapsed in 1935.

More than 100 people died.

The Gleno Dam which collapsed in 1923.

An estimated 350 people died.

https://damfailures.org/

https://pmc.ncbi.nlm.nih.gov/articles/PMC4997708/

17

I haven't tried to compile a global list of the worst hydro electric dam collapses, as this sort of information is actually very difficult to find, even on web sites dedicated to dam failures.

An additional problem is that information on whether a dam was used for electric power generation or not is often not available.

18

Dam failures where contradictory or insufficient information is available on whether there was an associated hydro power plant include the 1975 Banqian Dam failure, where death estimates range up to a quarter of a million.

19 Solar Panel Slow Accumulation

Contrast this with roof top solar panels.

Many small accidents can add up to big numbers as well.

20

Health and safety literature discussing solar panel safety mention things such as

Falls from roofs.

Electric shock.

Arc flash (burns from electrical arcing).

Normal electrical safety procedures which are based around locking out sources of energy do not work with solar panels which makes safety more difficult.

Heat stress due to working exposed in the hot sun.

Warning from US government on falls by solar panel installers.

https://stacks.cdc.gov/view/cdc/228946

https://www.osha.gov/green-jobs/solar

21 Why We Cannot Compare the Two

Hydro catastrophes are not well documented, but we can at least find records of some of the most notable ones.

However, even those have very large variations in estimates of deaths.

22

Roof top solar deaths however are largely undocumented.

The industry is largely unregulated.

There is no central authority which accumulates many individual deaths or injuries.

At best there are worker and public safety bodies who simply accumulate those statistics into general construction or household injuries.

23

Thus we have no reliable means of comparing the two energy sources on a comparable basis.

We face the same problem with all other major electrical energy sources.

So far as I am aware, there are no peer reviewed scientific studies which compare the relative safety of all of the major electrical energy sources we are considering here based on actual numbers.

--------------------

24 Safety Risks

I will now try to list some the major hazards for each of energy sources we are considering.

There is however limited data available.

In many cases we just have reference to worker safety organizations as to what the hazards are.

I will not attempt here to put numbers to these here.

Categories

25 Coal, Oil, Natural Gas

The hazards are

Air pollution

Mining and oil field accidents

Pipeline explosions

Transportation accidents. These- move a lot of material so these are significant.

26 Hydroelectric

These include

Dam collapse

Drowning

27 Nuclear

These include

Radiation exposure

28 Wind

These include

Falls

Confined space deaths (there is not much detail on this)

Electric shock

Ice throws (that is, throwing pieces of ice off the blades)

This technology has a significant problem with people working alone which greatly increases risks associated with other dangers.

29 Solar

These include

Falls

Electric shock

Arc flash

Heat stress

30

I have not tried to cover all possible risks associated with each category, just the ones which each industry considers to be the risks they concern themselves with.

There does not exist any means by which risks of similar types are compared across different industries.

31 Reliability of Supply is Also Safety

In a completely electrified net zero society, reliability of supply is a safety matter.

People will die in very large numbers in cold climates if they do not have heat.

If we have no fossil fuels, we need to also consider how reliably does a grid based on any of the options work.

I have not seen anyone attempt to address this question and will not attempt to address it here.

However, it must be addressed in any comprehensive attempt to rank safety.

--------------------

32 Studies or Articles on Estimates of Relative Safety

Despite the difficulties of comparing the safety of different sources of energy, some people have attempted this anyway.

Different estimates done at different times had different focuses, so unfortunately we do not have a nice set of studies that we can neatly use to cross check one another.

I will however list the names and the authors and summarize the results.

--------------------

33 The Health Hazards of Not Going Nuclear

By Dr. Petr Beckman

Published in 1976

The author of this book tried to address the relative safety of different sources of energy in the mid 1970s.

However, it is old at this point, so I won't bother digging through its pages to find his figures.

34

He mainly focused on comparing electric power generated with coal to nuclear.

His conclusion was that if the goal was to prevent deaths or ill health in the process of generating electricity, then the logical conclusion was to replace coal fired power plants with nuclear.

35

The book was relatively well known at the time, as least as far as books on energy are concerned, so I thought it was still worth mentioning.

I happen to have a copy of this book which I bought back in that time period

It was the 8th printing of the book, so it would appear to have had relatively good sales.

36

The author did address the issue of what I have termed "catastrophism" in his comparison of different energy sources, although I don't know if he used this phrase.

I don't know if he was the first to use this sort of analysis, but he certainly was very influential in terms of popularizing it.

--------------------

37 Risk of Energy Production

by Herbert Inhaber

Publication AECB 1119

March 1978

This study is a scientific paper from the same time period as the book "The Health Hazards of Not Going Nuclear".

38

He based his risk estimates largely on estimates of the amount of material which was used in the construction and operation of various power sources.

While we could argue over whether or not this is a valid methodology, I think any such argument would be pointless as I think the age of the study alone renders it not relevant today anyway.

Advancements in materials have changed the basis results significantly by now.

However, as it exists I thought I would mention it to show that the idea of comparing energy sources to each other is not a new one.

The author compared a wider variety of potential sources than Beckman did.

39

Here's his conclusions.

He assumes equal amounts of energy produced by each method.

The numbers are normalized such that the total sums to 100%.

You can think of it in terms of what proportion of total deaths or injuries would result from each source if each were equally used.

40

Coal 27.5%

Oil 25.6%

Methanol 16.7%

Wind 10.8%

Solar photovoltaic 9.2%

Thermal 8.1%

Solar space heating 1.5%

Ocean thermal 0.4%

Nuclear 0.13%

Natural Gas 0.08%

41

His natural gas estimate is drastically different from that of other authors.

I am not going to worry about explaining it however, as the study is as I said old enough to be not very relevant anyway.

I am mainly including this here out of historical interest.

42

As a footnote, the methanol he refers to would be synthesized from wood. This was a popular idea in that era as a means of providing liquid fuels for transportation. Practical battery electric cars in those days were strictly science fiction.

43

The ocean thermal category is a real blast from the past and I had forgotten all about that concept.

It was a very popular idea at that time and was supposed to be *the* big and upcoming thing in renewable energy.

It involved various means of attempting to extract energy from differences in water temperature at different depths in the ocean.

It gradually faded away however, as despite great efforts being put into it, designs never proved to be practical.

--------------------

44 Electricity generation and health

Anil Markandya, Paul Wilkinson

Published in the Lancet, Vol 370, 15 September 2007

45

This is more recent than the previous one, although it is nearly 20 years old at this point.

Unfortunately it doesn't cover wind or solar, just fossil fuels and nuclear.

However it is still useful, and the Lancet is a very reputable peer reviewed journal.

46

I will present just the results rather than discussing the whole paper.

The authors break it down into deaths among the public, occupational deaths, and air pollution related deaths, serious illness, and minor illness.

47

They break the energy sources down into lignite, coal, gas, oil, biomass, and nuclear.

Lignite is a type of very low grade coal used mainly for electric power generation.

In this paper biomass refers to energy crops and forest residues.

48

I will summarize the results by category rather than trying to describe a table that has 6 rows and 5 columns.

All numbers are normalized in terms of deaths or cases per TWh.

49

Occupational deaths from accidents

lignite 0.1

coal 0.1

gas 0.001

oil no data

biomass - no data

Nuclear is 0.019.

50

Deaths among the public from accidents

lignite 0.02

coal 0.02

gas 0.02

oil 0.03

biomass no data

Nuclear 0.003

51

Air pollution deaths

lignite 32.6

coal 24.5

gas 2.8

oil 18.4

biomass 4.63

Nuclear 0.052

52

Air pollution serious illnesses

lignite 298

coal 225

gas 30

oil 161

biomass 43

Nuclear 0.22

53

Air pollution minor illnesses

lignite 17,676

coal 13,288

gas 703

oil 9,551

biomass 2,276

Nuclear no data

54

Natural gas edges out nuclear power slightly in terms of occupational safety, but in every other category nuclear is drastically lower in terms of ill effects than any of the alternatives.

--------------------

55 2020 Fatalities for US Roofers Increased 15% as Solar Roof Installations Increase

Published in The Next Big Future

July 6, 2021 by Brian Wang

56

This seems to be written by someone who has a popular science blog.

I'm not familiar with it personally, but he addresses the subject so I'll list it.

The title implies that it's all about rooftop solar, but he provides comparative numbers for the other energy sources of interest, so that is useful for our purposes.

However, he doesn't describe his methodology, so we need to treat them with some caution.

Here are his results

These are deaths per thousand terawatt hours.

57

Coal - 100,000

Oil - 36,000

Natural gas - 4,000

Hydro - 1,400

Rooftop solar - 440

Wind - 150

Nuclear - 90

58

If we plot these numbers on a bar chart, coal and oil are so large that all of the others are squished to the bottom of the chart and are difficult to see at all.

Let's therefore look at these in terms of orders of magnitude.

Keep in mind that this is a logarithmic scale.

This means that the difference between 4 and 5 is much greater in linear terms than the difference between 1 and 2.

59

Coal - 5

Oil - 4

Natural gas - 3

Hydro - 3

Rooftop solar - 2

Wind - 2

Nuclear - 1

60

Each of these numbers represents an order of magnitude, that is a power of ten.

We can see that with rooftop solar, wind, and nuclear, the numbers are so close and the uncertainties are so great and their relative values so small compared to say coal that they can be seen as equivalent so far as safety is concerned.

--------------------

61 What are the safest and cleanest sources of energy?

by Hannah Ritchie

Published in Our World in Data

First published in 2017, updated in 2022 and 2024

62

The author of this study addressed both deaths and greenhouse gas emissions.

Deaths from accidents and air pollution are normalized to per TWh of electricity, while greenhouse gas emissions are normalized to GWh of electricity over the life cycle of the plant.

63

Here are the death figures.

Coal 24.6

Oil 18.4

Biomass 4.6

Natural Gas 2.8

Hydro power 1.3

Wind 0.04

Nuclear 0.03

Solar 0.02

64

For greenhouse gas emissions the figures are

Coal 970 tons

Oil 720 tons

Natural gas 440 tons

Biomass 78 to 230 tons

Solar 53 tons

Hydro power 24 tons

Wind 11 tons

Nuclear 6 tons

65

If we take the death figures and rank them by order of magnitude as we did with the previous article, we get the following.

66

Coal - 4

Oil - 4

Biomass - 3

Natural Gas - 3

Hydro power - 3

Wind - 1

Nuclear - 1

Solar - 1

67

Keep in mind that the previous article covered only rooftop solar and not large industrial installations, and so is not directly comparable.

Also the units are different, with the previous article being in terms of thousand TWh, and this one being in TWh.

If we exclude solar (as the numbers are not comparable), Brian Wang's numbers are between 1.5 to 4 times higher than Ritchie's, except for hydro which are almost identical. I think this latter is due to both sets of numbers are dominated by one exceptionally big hydro accident.

68

Overall however, the relative rankings are quite comparable.

Ritchie's numbers for deaths from coal, oil, and natural gas appear to be directly from the study by Markandya and Wilkinson mentioned above.

For the benefit of those who are wondering, Ritchie specifically states that her numbers for nuclear include the Chernobyl and Fukushima accidents.

--------------------

https://www.iaea.org/publications/magazines/bulletin/21-1/solar-power-more-dangerous-nuclear

Direct link to file

https://www.iaea.org/sites/default/files/publications/magazines/bulletin/bull21-1/21104091117.pdf

https://ourworldindata.org/safest-sources-of-energy

https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(07)61253-7/abstract

https://www.nextbigfuture.com/2021/07/2020-fatalities-for-us-roofers-increased-15-as-solar-roof-installations-increase.html

--------------------

69 Conclusion from Studies

Remember that in engineering terms, when comparing groups of numbers which contain both both very small numbers and one or more very large numbers, the differences between the small numbers are often not significant.

The differences between the small numbers may be the product of our ability to measure these things rather than any real differences.

70

For example, in the article by Ritchie wind power would appear to be twice as dangerous as nuclear.

However, the difference between them is 0.02 compared to 24.6 for coal.

In other words, the difference between apparently "dangerous" wind and apparently "safe" nuclear is equivalent to 0.08% of the total for coal.

It's therefore meaningless and a red herring to even worry about.

71

With the above taken into consideration, generally the different sources of energy fall into two broad categories in terms of number of deaths, injuries, and illnesses.

The fossil fuels and biomass fall into one group and wind, solar, and nuclear into another group.

72

Hydro power would seem to fall into the higher risk category or at least somewhere between the two, but this I suspect is mainly due to one exceptionally large dam collapse in China, the Banqian Dam failure in 1975.

This is mentioned as being specifically included in the article written by Ritchie.

This was a multi-purpose dam, and information on this dam is difficult to find.

It is not clear to me whether it had a hydro electric generator associated with either it or another dam that was part of the same system.

73

Some people therefor may argue for its exclusion from the numbers.

Of course some people may argue for its inclusion anyway, as it was a dam regardless of whether it actually had an electric generator attached.

If we exclude it, then I think the numbers for hydro power would fall into the same range as for nuclear, wind, and solar.

74

Most people would consider hydro power to be safe and clean enough regardless of this and I will rank it as such in any conclusions that I come to.

As you can see, even if we have numbers, it can be a matter of opinion as to how to interpret them.

--------------------

--------------------

75 Taking a Systems Approach

Now let's take a look at the broader energy picture today and into the future.

Many countries in many parts of the world have committed to the concept of "Net Zero", which means eliminating carbon emissions on a net basis.

Net zero essentially means the complete electrification of society.

We must therefore have electrical energy on demand and at low cost.

We must as a result of this look at complete electrical systems rather than individual sources in isolation.

76

At one time many electrical systems were entirely coal or entirely hydroelectric.

This is no longer the case.

There are now major amounts of wind and solar involved in many countries.

However these are inherently intermittent.

This means that other sources of energy are inherently also required to have a functional system.

77

If any particular solution inherently requires fossil fuels to meet part of the demand, then the safety, pollution, and climate issues relating to those fossil fuels have to be factored in to that complete system when trying to come up with a relative ranking.

Talking about Individual sources in isolation are therefore meaningless in these countries.

78

There are battery systems, but these are mainly used to stabilize and regulate the grid plus to a lesser degree to smooth out short term daily peaks in demand.

They do not have the ability to store large amounts of electricity on a large scale for an entire grid for days, weeks, and months to make up for intermittency.

79

So a serious attempt to rank sources of energy would need to look at a variety of representative countries and for each one come up with a plan that involves 'x' megawatts from source 'a', 'y' megawatts from source 'b', etc., and total up the values for each.

80

I am not aware of anyone who has studied this larger issue.

However, the problem has to be addressed from this perspective in order for any answer to be useful.

Not taking this into account is like ordering a diet soft drink to go with with a high calorie meal and assuring yourself that your plans to diet are fine.

81

This is not to imply there is anything inherently wrong with wind or solar.

It does mean that if your goal is to achieve both net zero and a clean environment, you have to look at your entire energy system as a complete system rather than focusing on what you feel are the most reassuring parts of it while ignoring the rest.

This does however add to the argument that it is in fact inherently very difficult to come up with a system of ranking energy sources for safety.

--------------------

82 Nuclear, Climate, and Clean Air - Contrasting Examples

To give a tangible example we will now look at two different places that followed two divergent paths at roughly around the same time frame.

These are the province of Ontario in Canada, and Germany.

83

Ontario had a mix of coal, hydro electric, and nuclear generating plants.

Germany had a mix of coal, nuclear and natural gas plants.

Ontario shut down their coal fired plants and kept their nuclear plants.

Germany however shut down their nuclear plants and kept their coal fired plants.

84 The Phase Out of Coal in Ontario

In 2003 Ontario decided to close all of its coal fired generating plants, which consisted of 19 units (that is boilers and turbines) totalling 8,800 MW.

This phase out was completed by 2014.

85

Here are the figures for amount of power generated by each energy source in 2003 and 2014.

Nuclear went from 42% to 60%

Hydro went from 23% to 24%

Gas went from 11% to 9%

Coal went from 25% to 0%

Non-hydro renewable went from 0% to 7%.

86

As you can see, the bulk of that replacement came from increased use of nuclear power.

Furthermore, this did not result in simply replacing coal with natural gas.

While gas is cleaner than coal, it still has emissions and if you recall from the studies that we looked at earlier, had an estimated death rate roughly 2 orders of magnitude greater than nuclear, solar, or wind.

87

To put this in more practical terms, at one time Toronto regularly had clouds of smog obscuring it, to a large extent due to these coal fired power plants

With the phase out of coal, smog days went to zero in 2015 compared to 53 a decade earlier.

The 2023 figures for Ontario show carbon emissions of 53 grams per kWh of electricity generated.

We can use this as a rough benchmark comparison for total emissions.

88 The Phase out of Nuclear in Germany

Until March of 2011, Germany generated one quarter of its electrical power from nuclear.

Starting in 2011 however, they began shutting down their nuclear power plants.

These were then phased out over the next decade.

However, the coal plants were to be kept to 2038.

In 2026 Germany began talking about increasing use of coal in order to save gas.

In the same year the German chancellor Friedrich Merz stated that the phase out of nuclear was a

quote “serious strategic mistake”.

EU Commission President Ursula von der Leyen said it was "a strategic mistake for Europe to turn its back on a reliable, affordable source of low-emissions power".

89

I won't go into the details of the phase out, but let's look at some emissions numbers for Germany.

If we look at the official numbers from the European Environmental Agency for 2024, for Germany their emissions were 298 grams per kWh of electricity generated.

Recall that we are using emissions as a very rough guide to amount of air pollution, and that this has a direct effect on the safety of the overall electrical energy system.

90

So, who actually made their people safer, Ontario who phased out their coal plants and kept their nuclear plants, or Germany who phased out their nuclear plants and kept their coal plants?

91

If you want a comparison directly within Europe, then Germany has one of the highest rates of emissions per kWh of electricity generated, whereas France, who use mainly nuclear power, have one of the lowest at 43 grams per kWh of electricity generated.

Again, who is making their people safer, Germany or France?

92

I don't want to make it sound like I am picking on Germany.

I am also not going to tell them how they ought to run their country.

However they provide a good real world example of how we need to look at things in overall context when we are thinking about the choices that we make.

https://www.ontario.ca/page/end-coal

https://www.cbc.ca/news/canada/windsor/smog-study-shows-significant-decreases-in-pollutants-in-ontario-1.4151183

https://www.eea.europa.eu/en/analysis/indicators/greenhouse-gas-emission-intensity-of-1

https://world-nuclear.org/information-library/country-profiles/countries-g-n/germany

https://www.politico.eu/article/friedrich-merz-is-right-to-reject-germanys-nuclear-phase-out-says-iea-chief-fatih-birol/

https://www.politico.eu/article/germany-considers-ramping-up-coal-power-to-avert-energy-crisis/

https://www.iea.org/countries/estonia/electricity

https://www.iea.org/countries/malta/electricity

--------------------

93 Conclusions

As we can see, there don't appear to be an abundance of peer reviewed scientific studies that we can simply point to in order to answer the question of safety of all possible major different energy sources once and for all.

Collecting the data to even attempt to answer the question is inherently very difficult as we cannot readily conduct experiments to answer the question, and sources of data are not collected or consolidated in a manner which can answer this question adequately.

94

The essence of the problem is that most energy industries are not as tightly regulated and monitored to the same degree that say nuclear power or commercial airliners are, so this data is simply not being systematically recorded.

However, a number of people have attempted to make estimates.

95

Their conclusions would seem to be that nuclear, wind, and solar are roughly equivalent in terms of safety.

All fossil fuels are much less safe than nuclear, wind, and solar, by as much as several orders of magnitude.

96

We can however say with a reasonable degree of certainty that if a country shut down their nuclear power plants and kept their fossil fuel plants, particularly coal, then they probably made their people less safe than if they had done things the other way around.

97

I hope that I have provided some context in which to think about the issue.

Thanks again to brian in ohio for providing the question upon which this episode is based.

--------------------

Provide feedback on this episode.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.

The echo command is very useful—it prints the arguments given to it, followed by a newline character. (The newline is sometimes also called a linefeed character depending on who is writing or speaking, and has the ASCII decimal value 10.) It has many uses, either in a script or interactively on the command line. The echo utility is used to display text, the value of a variable, or the result of a pathname expansion. It can also feed text to another command in a pipeline.

As useful as echo is, it should come as no surprise that it first appeared early on in Bell Laboratories' Second Edition UNIX ¹ in 1972. This initial version accepted no options ² —although the manual page doesn't explicitly say output is followed by a newline character, the description of writing "as a line" seems to imply it. In Seventh Edition UNIX, the manual page ³ makes that clear, and also features the addition of the -n option, which causes echo to print the arguments without a trailing newline character. Eighth Edition UNIX's echo ⁴ gained the -e option, which allows certain escape codes from the C programming language to be used.

These variations caused differences in behavior between different versions of echo . Will running echo -n something on your system output the text "something" without a newline, or "-n something" followed by a newline? Things get even trickier when the command arguments include parameter or pathname expansions. If there are files named "-n" and "something" in the current directory, what does echo * output? Like the previous question, that depends on whether or not your version of echo treats -n as an option. You can't get around this ambiguity by quoting or escaping the "*", because that just causes echo to print a literal asterisk.

Example using GNU utilities on Debian 12; both the "echo" utility and the "echo" builtin of bash recognize "-n" as an option.

$ ls -1
-n
something
$ echo *
something$ #Shell prompt is on the same line because "-n" was treated as an option to echo
$ echo "*"
*

The solution was to create a new utility, which is the first UNIX Curio for today: printf . This command allows a user to print text similar to the way the identically-named function works in the C programming language. You run printf ⁵ followed by a format string, followed by zero or more arguments. No newline characters are printed unless specifically indicated by the format string or the arguments.

To use printf to print "something" without a newline, that would just be printf something . This demonstrates that you don't need any arguments—in this example, the format string is just a set of regular characters to be displayed. If you wanted a newline character at the end, printf "something\n" would give you that. (In this case, the format string needs to be quoted so the "\n" isn't interpreted by the shell.) In addition to "\n" for a newline, you can also use "\a" for an alert (rings the terminal bell), "\b" for a backspace, "\f" for a formfeed, "\r" for a carriage return, "\t" for a horizontal tab, "\v" for a vertical tab, and "\\" to get a literal backslash. In addition to these special characters, any arbitrary byte can be included using a backslash followed by one to three octal digits; however, it might be difficult to predict what will be output because it can differ based on the character set the terminal is using. It is safer and more portable to stick to the pre-defined characters if possible.

The real magic of the printf utility comes from using "conversion specifications" in the format string. Probably the simplest of these to explain is the "%s" conversion specification—it represents a string of any length. The command printf "Hi, %s, how are you?\n" followed by a list of names would print the greeting for each name, putting it in the place occupied by the "%s".

$ printf "Hi, %s, how are you?\n" Alice Bob Carol
Hi, Alice, how are you?
Hi, Bob, how are you?
Hi, Carol, how are you?

The format string is reused as many times as needed to consume all of the arguments. Take, for example, the command printf "Hi, %s, have you met %s?\n" . If this is run with two name arguments, it would print the sentence on one line, using both names. If run with four name arguments, it would print the sentence twice, once with the first two names and again with the second two names. If you only gave it three names, the last "%s" conversion specification would be replaced with a null string.

$ printf "Hi, %s, have you met %s?\n" Alice Bob
Hi, Alice, have you met Bob?
$ printf "Hi, %s, have you met %s?\n" Alice Bob Carol David
Hi, Alice, have you met Bob?
Hi, Carol, have you met David?
$ printf "Hi, %s, have you met %s?\n" Alice Bob Carol
Hi, Alice, have you met Bob?
Hi, Carol, have you met ?

Three other items can also be given in each conversion specification: flags, the field width, and the precision. The exact meanings of these depend on which type of conversion specifier character you are using. For "%s", using a "-" as the flag causes the text to be left-justified instead of the default right-justified, a field width causes the printed field to be at least as long as the number given, and a precision limits the number of bytes written from the string to the number given.

$ #Example of %s with a precision value
$ printf "Hi, %.3s, how are you?\n" Alice Bob Carol
Hi, Ali, how are you?
Hi, Bob, how are you?
Hi, Car, how are you?
$ #Example of %s with a field width
$ printf "Hi, %8s, how are you?\n" Alice Bob Carol
Hi,    Alice, how are you?
Hi,      Bob, how are you?
Hi,    Carol, how are you?
$ #Example of %s with a left-justify flag and a field width
$ printf "Hi, %-8s, how are you?\n" Alice Bob Carol
Hi, Alice   , how are you?
Hi, Bob     , how are you?
Hi, Carol   , how are you?
$ #Example of %s with a left-justify flag, a field width, and a precision
$ printf "Hi, %-8.3s, how are you?\n" Alice Bob Carol
Hi, Ali     , how are you?
Hi, Bob     , how are you?
Hi, Car     , how are you?

While "%s" is probably the most commonly-used conversion specification, others are available. A whole set of them are dedicated to printing integer values as a signed decimal, an unsigned decimal, an unsigned octal, or an unsigned hexadecimal number. These also can take flags, a field width, and a precision. I think the details and nuances of all this are too complex to clearly explain here, so I will just refer you to the POSIX "file format notation" specification ⁶ .

Be aware that unlike the printf function in the C programming language, the printf utility is not obligated to accept conversion specifications for floating-point numbers. While some implementations might support this, scripts intended to be portable should limit themselves to the restricted set required by the POSIX standard (%d, %i, %o, %u, %x, %X, %c, and %s, plus %b and %% described below).

Two more conversion specifications are worth mentioning. The first is only required by the standard for the printf utility, not the C function, and is "%b". This is the same as "%s", except that certain backslash escape sequences in the argument will be treated specially. This includes all the ones described above except for the one using octal digits to represent a byte. In an argument, this is instead represented by "\0" followed by one to three octal digits. An additional backslash escape sequence accepted is "\c"—this does not print anything itself, but causes printf to immediately halt output.

The final conversion specification is "%%", which just outputs a literal "%". You can't use a bare "%" in the format string, because printf expects that to introduce a conversion specification. Be careful not to be tripped up by this when trying to print some value as a percentage.

Example assuming that the hypothetical "/dev/batterycharge" file on your laptop outputs the battery charge level (42% in this case). As you can see, in some cases an error message might be displayed, but in others it might just behave in a way you didn't intend without complaining. GNU's "printf" utility and the "printf" builtin of bash both support "%e" as a conversion specification as an extension to POSIX.

$ cat /dev/batterycharge
42
$ #Wrong
$ printf "Your laptop's charge level is $(cat /dev/batterycharge)%.\n"
bash: printf: `\': invalid format character
Your laptop's charge level is 42$ #Shell prompt appears here from the error
$ #Right
$ printf "Your laptop's charge level is $(cat /dev/batterycharge)%%.\n"
Your laptop's charge level is 42%.
$ #Next one treats %e as the specifier, with the space and "l" as flags
$ printf "Your laptop has $(cat /dev/batterycharge)% level of charge.\n"
Your laptop has 42 0.000000e+00vel of charge.
$ #Because no arguments were given, "0" was used for the value to convert

Let's go back to the situation I was describing with echo —we have files named "-n" and "something" in the current directory and want to print all their names, separated by spaces. We could do that with printf "%s " * , which would not treat the "-n" as an option. However, the output might look a little weird because there wouldn't be a newline character at the end. We could insert a newline by using "%b" instead of "%s" and following the asterisk with a "\n\c" as the second argument. The "\c" is there to prevent the final space in the format string from being printed after the newline.

$ ls -1
-n
something
$ printf "%s " *
-n something $ #No newline was printed here
$ printf "%b " * "\n"
-n something
 $ #There's a newline, but also a spurious space before the shell prompt
$ printf "%b " * "\n\c"
-n something
$ #No space before the shell prompt this time

Using the "%b" conversion specification can therefore solve one problem, but it also introduces another. Arguments which include a backslash can be interpreted as escape sequences, and many systems are fine with allowing backslashes in filenames. In cases where you're just using the printf utility to display text, it's usually not a big deal if the output looks a little wonky. Where you really need to be careful is when the text is being piped to another program, as control characters and other oddities might cause unexpected results, and can potentially create security problems if processed by a script or utility running as a privileged user.

$ #GNU "ls" displays filenames containing a backslash in single quotes
$ ls -1
apple
banana
'\cherry'
durian
$ printf "%b " * "\n\c"
apple banana $ #"\c" in "\cherry" stops output immediately

The printf utility looks to have shown up first in 1986's Ninth Edition UNIX ⁷ , though the earliest manual page I could find ⁸ is from the Tenth Edition. Its first appearance in BSD seems to be from 1990 in the 4.3 Reno release ⁹ . Two years later, it was added to Issue 4 of The Open Group's CAE Specification. From what I can tell, it did not seem to be in AT&T's System III—presumably the printf utility did make it into System V at some point but I found it difficult to track this down.

While echo is still suitable for use where you know for certain that you want a newline character printed at the end and none of the arguments will start with a hyphen, consider using the printf utility instead for displaying text. It offers more flexibility and features than you are guaranteed to get with echo , although it does require a bit of forethought in constructing a proper format string and arguments. That is not necessarily a bad thing, because a script's author should be thinking about what might happen if it is called with "strange" text or filenames.

This episode also provides a good case for being careful when naming files—many filesystems will allow you to use hyphens, control characters, quotation marks, and potentially any character other than a slash or a null byte in a filename. As we've seen, some of these characters can create problems for standard utilities. While it can feel limiting, especially for people not using English, the safest filenames to use on a UNIX-like system consist only of characters in the "portable filename character set" as defined by POSIX ¹⁰ and where the first character is not a hyphen. This set includes the lowercase and uppercase letters "a" through "z", the numerals "0" through "9", and the period, underscore, and hyphen. Notably, it does not include the space character.

That leads me to another UNIX Curio that I only just now discovered while researching this episode. This is the pathchk utility ¹¹ . It can be run with one or more strings as arguments, checks each one against a set of rules for pathnames, and outputs an error message for each problem found. By default, it checks against the following limits on the system where it's being run: maximum number of bytes in the full path, maximum number of bytes in any component of the path, all byte sequences must be valid in the given directory, and the user running the program must have access to all directories referenced. If run with the -p option, instead of those limits, it checks against POSIX limits: a maximum of 256 bytes in the full path, a maximum of 14 bytes in each component of the path, and each component must only include characters from the portable set. The -P option adds warnings if any component starts with a "-" or if the pathname is completely empty. While the exit status will tell you if the checks succeeded or not, I don't feel like the pathchk utility is well suited to be used in an automated fashion, as the exact wording of its output is not specified and checks cannot be selected individually. However, it can be used interactively to validate pathnames you aren't sure about. See the linked specification for full details.

References:

1. A Research UNIX Reader: Combined Tables of Contents https://archive.org/details/a_research_unix_reader/page/n99/mode/1up
2. A Research UNIX Reader: Second Edition UNIX echo manual page (although this page has "v1" typed at the top, the date and the tables of contents indicate it first appeared in v2, a.k.a. Second Edition) https://archive.org/details/a_research_unix_reader/page/n22/mode/1up
3. Seventh Edition UNIX echo manual page https://man.cat-v.org/unix_7th/1/echo
4. Eighth Edition UNIX echo manual page https://man.cat-v.org/unix_8th/1/echo
5. Printf specification https://pubs.opengroup.org/onlinepubs/9699919799/utilities/printf.html
6. File Format Notation specification https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap05.html
7. A Research UNIX Reader: Ninth Edition Table of Contents https://archive.org/details/a_research_unix_reader/page/n95/mode/1up
8. Tenth Edition UNIX echo/printf manual page https://man.cat-v.org/unix_10th/1/echo
9. 4.3BSD Reno printf manual page https://man.freebsd.org/cgi/man.cgi?query=printf&sektion=1&manpath=4.3BSD+Reno
10. Definitions: Portable Filename Character Set https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap03.html#tag_03_282
11. Pathchk specification https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pathchk.html

Provide feedback on this episode.

I started out with Basic on the TI-99/4A in 1984. The bare machine could not be programmed by the user in machine code. In 1985 I bought a ZX Spectrum, that gave me total control over the machine.

I wrote two FORTH systems on the ZX-Spectrum.

In 1988 I got my first 8088 PC, also programming it in FORTH.

In 1992 I got an 80386 PC and I ran Linux on it. MCC Interim Release from v. This was the first Linux distro.

I have been using Linux ever since.

From then on I obtained newer PCs, such as a Pentium in 1995, a Pentium-2 in 1998, a Pentium-4 in 2003 and a Core-2 Duo in 2006.

I used several Linux distributions: but I always return to Debian.

Links:

• https://github.com/ForthHub/F83 F83.COM is the ready to run FORTH system.
• https://github.com/uho/F-PC F-PC - a Forth system optimized for IBM-PC, XT and AT machines running DOS
• https://www.latte.org/latte.htmlLatte The Language for Transforming Text
• https://en.wikipedia.org/wiki/Joe's_Own_Editor
• https://www.freebsd.org/
• https://www.debian.org/
• https://www.gentoo.org/
• https://en.wikipedia.org/wiki/Mac_Mini
• https://en.wikipedia.org/wiki/PowerPC
• https://en.wikipedia.org/wiki/Ivy_Bridge_(microarchitecture)
• https://en.wikipedia.org/wiki/Ubuntu
• https://en.wikipedia.org/wiki/Firefox
• https://en.wikipedia.org/wiki/Chromium_(web_browser)
• https://www.raspberrypi.com/products/raspberry-pi-400/
• https://www.intel.com/content/www/us/en/ark/products/series/217838/12th-generation-intel-core-i5-processors.html
• https://en.wikipedia.org/wiki/Windows_11

Provide feedback on this episode.

Pokémon Go (stylized as Pokémon GO) is a 2016 augmented reality (AR) mobile game developed and published by Niantic, in partnership with Nintendo and The Pokémon Company, for iOS and Android devices. The game uses GPS to locate, capture, train, and battle Pokémon. It is free-to-play, featuring a freemium model that includes local advertising and offers in-app purchases for additional in-game items. Pokémon Go launched with approximately 150 Pokémon species, with new species regularly introduced.

-- Wikipedia.org

Provide feedback on this episode.

Some links to other tutorials (in case you need a visual element, these helped me out)

Mixing board tutorial

https://youtu.be/H-7kQmpjBds

Drone Box tutorial

https://youtu.be/50C6DBsqy24

Provide feedback on this episode.

T-DOSE

TDOSE 2027

Mark you calendars #TDOSE 2027 on 5 and 6 June '27 in the Weeffabriek, Geldrop.

• T-DOSE
• Info Booth
• Hackalot
• Laptop Revive
• Free Software Foundation Europe
• Doeidag and Banray
• Debian
• Angry Nerds Podcast
• Freie Software Freunde - Free Your Model Train
• Hacker Public Radio: The community Podcast
• UBports
• Adfinis
• Credits

The Technical Dutch Open Source Event (T-DOSE)

In hpr4641 :: Technical Dutch Open Source Event (T-DOSE) , Ken interviewed Peter van Ginneken about the T-DOSE conference.

The Technical Dutch Open Source Event (T-DOSE) is a free conference to promote the use and development of Open Source software. This event has is organised yearly since 2006 in the Brainport region, near Eindhoven, The Netherlands. During this event, Open Source projects, developers and visitors can exchange ideas and knowledge.

Peter van Ginneken Opens the Event.

We catch up with him at the start of Day 2.

Info Booth

The backbone of any event is the Info booth and catering.

Here we talk to Nick Hibma who when not serving on the Info Booth is treasurer of the T-DOSE organisation.

Ready to serve sandwitches, sell T-Shirts, Magic Mugs, and club-mate

T-Shirts

club-mate

Magic Mugs

Hackalot

Hackalot is the Eindhoven and surrounding area hackerspace. A hackerspace is a place where hackers can work on their own or collaborative projects. You can work and talk together, but you can also do your own thing. Together we can also purchase a lot of cooler tools such as lasercutters and 3d printers. Often there is no suitable place for equipment at home. So if you know someone, you are either an electronics/computer/technical hobby that got out of hand, come on by!

Boekenwuurm at the Hackalot stand.

The Hackalot stand.

• Boekenwuurm@hsnl.social
• boekenwuurm.nl
• Hackalot

Laptop Revive

Laptop Revive collects discarded laptops, that are still working. We then install Linux Mint to provide a working laptops to students who cannot afford laptops. We are socially involved, sustainable and open.

Alex Kok Laptop Revive

• Laptop Revive

Free Software Foundation Europe

Free Software Foundation Europe (FSFE) information booth, with information material, stickers and merchandise.

Nico was so busy that we were unable to snag an interview this time. However check out our talk with him at the NLUUG Spring Conference 2026 .

• Free Software Foundation Europe

Doeidag and Banray

We also interviewed Geert-Jan Meewisse in hpr4639 :: NLUUG Spring Conference 2026 but this time he is here talking about banray.eu

In 2025, Meta sold over seven million pairs of camera-equipped glasses that look like regular Ray-Bans. The person wearing them looks like anyone else. But these people are now products, as is everyone they interact with.

He then also mentioned the Doeidag project where they encourage people to drop one service at a time on the first Sunday of the month

• https://doeidag.nl/
• https://banray.eu/

Geert-Jan Meewisse Doeidag and Banray

Debian

The Debian Project is an association of Free Software developers who volunteer their time and effort in order to produce the completely free operating system Debian.

Ken Talks to Joost van Baal Llić from the Debian Project

Debian

Angry Nerds Podcast

Angry Nerds, met extra cyber!

The Angry Nerds is a Dutch Language podcast about privacy and security

It's a live show that is topical and often humorous tech podcast where a group of enthusiastic nerds discusses current technology, IT and cybersecurity topics. The hosts combine technical depth with background conversations, humor and the occasionally a good dose of cynicism. Expect conversations about everything from network infrastructures to software development, from privacy issues to bizarre tech trends.

Ken on the Angry Nerds Podcast

You can listen to the recording at Angry Nerds op T-DOSE 2026 deel 2 (prikkelarme versie).

• Angry Nerds Podcast

Freie Software Freunde - Free Your Model Train

We are a non-profit organization. We are committed to Free Software and Open Standards. Software is not just technology, it's an important part of our daily life.

We want to raise awareness of the importance of Free Software and Open Standards. That is why we are concerned with topics outside of technology: politics, education, ethics, psychology, ecology and economics, licenses, ... One of our projects is "Free your model train". Our goal is to raise awareness of the benefits of open standards.

Birgit Hücking (@akkolady) standing at the freie-software.org

The freie-software.org table with two large train loops, a smaller internal one. Two knitted Tux Mascots. And a lot of information.

Close up of the two knitted Tux Mascot.

• @akkolady@chaos.social
• @FreieSoftwareFreunde@mastodon.social
• Freie Software Freunde
• Free Your Model Train
• https://fymt.de

Hacker Public Radio: The community Podcast

Hacker Public Radio is a technology focused podcast that releases shows every weekday Monday to Friday. Our shows are created by people like you, and can be on any topic that is of interest to hackers, hobbyists, makers, etc. We are a welcoming community that offers positive feedback and encourages respectful debate. This is our 21st year of operation, and we will release our 5,000th show in August. Everything we do is released under a Free Culture License. We do not vet, edit, moderate or in any way censor any of the audio you submit, we trust you to do that. We will be available to guide you in sharing your knowledge with the community. Having had a stand at FOSDEM (BE), OggCamp(UK), Linux Fest North West(US), Spectrum (FR), we are available to show you how easy podcasting can be. We will be answering your questions, and conducting interviews with anyone with anything interesting to say.

The HPR booth.

• Hacker Public Radio

UBports

We are developing an open source Linux mobile OS built to be your daily driver... ...and we'd like to welcome you to our community.

Next up is a chat with Sander Klootwijk about UBports and Ubuntu Touch. Their website has a list of supported devices .

We talk with Sander Klootwijk

Proof it's running on actual hardware

Yumi The UBports Installer Mascot was not available for comment.

Ubuntu Touch on a Fairphone

• @BallonQuartier@mastodon.nl
• UBports
• https://devices.ubuntu-touch.io/

Adfinis

Accelerate your business with open source-driven automation, security, cloud, and DevSecOps solutions from Adfinis, your end-to-end partner for robust, flexible IT that drives growth and innovation at any scale. Welcome to Our World Full of Open Source

At Adfinis, we believe in the transformative power of open source technology to foster innovation, transparency, and collaboration. We are committed to providing solutions free from vendor lock-in, ensuring our clients retain full control and flexibility over their systems. Digital sustainability lies at the heart of our approach, as we strive to create technologies that not only serve the present but also support a long-term, environmentally responsible future. Additionally, we champion digital sovereignty, empowering organizations and communities to own and control their data, infrastructure, and technological destiny. These principles drive us to build a more open, sustainable, and inclusive digital world.

Finally we chat to Coen hamers , Robert de Bock , and Annebelle van Waardenburg from Adfinis whose sponsorship made the event possible.

• https://www.adfinis.com/en/solutions
• https://www.adfinis.com/en/career

Credits

• Record Needle Rip
• Free Software Song

Provide feedback on this episode.

Wandering along the suggestion of topics in Hacker Public Radio, we come to this: how to obtain the answer to something that I don’t even know how to ask?

My opinion is: you can’t get an answer without the desire, be it represented by a question or a quest.

So, how to pose the correct question? There are a few steps!

1st, interest . Get motivated to know, even if you don’t know exactly what you want specifically.

2nd: be open to deconstruction. To learn is to recognize you didn’t know before (self-pride has to give place for learning). And when confronted with the new, things you thought you knew — even ones you relied on before — may need to be now rejected, if incompatible with the truth you only now receive.

3rd: when the barriers are down, it’s possible to study the theme. Because, at the moment, you don’t know what you don’t know; while one ignores something, he’s only ignorant. One have to listen to others, to practice and /or reflect on the subject until he reaches something specific (and not easily or promptly answerable) worth being the object of his quest or question.

4th: with the question in hand, go for the answer. If it’s valuable for you, more time or less time you are definitely going to get it. If it’s too broad, or you lose interest, or if you need more after getting the initial answer, no problem: you can reformulate and start from the beginning. Great part of the necessary questions only come after the answer already came by a long course of inquiries [questions] (natural consequence of studying with a desire).

* * *

Credits:

1) Writing is putting thoughts in order and being able to detect what is loosen or unfounded (needing refinement of discard). “The reason so many people have trouble writing is that it’s fundamentally difficult. To write well you have to think clearly, and thinking clearly is hard.” ( https://paulgraham.com/writes.html )

2) “What Are the Answers I Need, To the Questions I Don't Know Enough to Ask?” ( https://hackerpublicradio.com/about.html#requested_topics )

( 1 and 2 came at a particular time together and where more deterministic to the line of this episode. )

3) Teacher Roosevelt, a course administrator (coordinator) of my post-graduation at Escola Paranaense de Direito, that ministered about the end of course paper and how to pose the question you are going to pose in order to write about a topic.

4) Gym exercises (because doing seated leg curls is difficult and gives strange sensations to many men; as thinking , gives a strange feeling one can’t even discern immediately if it’s good, bad, or only odd).

Sound effect for credits numbering: “Number pad 1”, by phatkatz4 on Freesound ( https://pixabay.com/sound-effects/musical-number-pad-1-105399/ )

Thank you!

Provide feedback on this episode.

• Blog post: https://claudiomiranda.wordpress.com/2026/05/19/laptop-computer-woes-or-how-i-learned-to-love-my-tech-hoarding/
• HP ProBook 4540s specifications: https://support.hp.com/au-en/product/product-specs/hp-probook-4540s-notebook-pc/
• Tonymacx86 forum post on the 1920x1080 screen upgrade for the ProBook 4540s: https://www.tonymacx86.com/threads/4540s-lcd-screen-1080p-replacement.88212/
• Panelook.com link to the LG156WF1 screen variants: https://www.panelook.com/modelsearch.php?keyword=LP156WF1&search_id=1839426
• HP ProBook 4540s teardown and screen upgrade video by JSnake Repair: https://youtu.be/3ceZ1m7JFYw?t=1252

Provide feedback on this episode.

In his early days as a writer, Heinlein wrote his stories in the context of a shared universe that he called the Future History. These were mostly short stories at first, with hte occasional novella. But they inclode some great stories.

The Future History, Part 1

One thing Heinlein became well known for was his Future History. This placed many of his stories in a common framework of a future environment, and allowed events from one story to influence events in other stories. Here is what he had to say about it, in a post I found on the Heinlein Society Facebook site:

“I never “created” or “invented” a “Future History.” On April Fool’s Day 1939 I started to write commercially; by the middle of August I had written 8 shorts & a serial. As 5 of these items were more or less to the same fictional background, I found that I was continually having to check back to keep from tripping over my own feet.

So I took an old navigation chart, about 3×4 feet, turned it over, made the time scale vertical, then set up 5 columns: stories, characters, technical data, sociological, remarks. Then I checked those first 5 stories, filled data into proper columns at the proper height for the fictional date—and continued to do this with other stories later. The chart was on the righthand wall near my elbow and was unusually messy as I never took the chart down to add to it—just reached over and scrawled on it.”

Source: https://www.facebook.com/HeinleinSociety/posts/i-never-created-or-invented-a-future-history-on-april-fools-day-1939-i-started-t/1092968002874634/

One thing that became clear as his Future History developed is that he was not looking at our future exactly. He was very clear in his mind that he was writing fiction, and not issuing prophecies. If you are reading it today, it is best to think of this as a kind of alternate timeline, and this is something that holds true through a lot of his work. Even in his later novels, which were never formally part of his Future History, he would mention events from that past group of works, which may implicitly incorporate them. But this is an area where scholars are in disagreement as to which if the later novels, if any, should be incorporated. And there were unwritten stories that appeared on the chart that would have given further background to the stories that were written. They were stories Heinlein seems to have intended to write at some point, but never got around to writing. You can get more information about this in his book Revolt in 2100.

The Future History stories were initially collected primarily in three books: The Man Who Sold the Moon (1950), The Green Hills of Earth (1951), and Revolt in 2100 (1953). Each of them fleshes out this hypothetical world in different ways. The first one, The Man Who Sold The Moon, introduces us to a businessman named D.D. Harriman, who is obsessed with going to the moon. But he thinks it should be done by private enterprise rather than by government. So he concocts a scheme to do this. He promotes a legal theory that the rights to the moon belong to the countries that it directly flies over, sort of like air rights taken to infinity. Then he uses the chaos of competing interests to throw this into the United Nations, and then gets the U.N. to give him the rights. He finally gets to launch a mission to set up a Moon base, but cannot join the expedition because the corporation considers him too valuable to risk. In a sequel story, Requiem, he does get to the moon just in time to die there. Heinlein was never above writing a tear-jerker.

Of course, the book has other stories not linked to D.D. Harriman. Heinlein’s fist story, Life-Line, is also collected here. And his second story, Let There Be Light, anticipates the development of solar power panels, but similarly to Life-Line, this earns the enmity of corporate interest, in the form of the Power Syndicate. The Roads Must Roll postulates moving roadways in the future, but the story really is about the sociology of technology in the future. And Blowups Happen, originally from 1940, anticipates nuclear fission as a power source, but it proves to be dangerous. They claim that the craters on the moon were really caused by a series of explosions to reactors that wiped out an earlier civilization. So they move the reactor into space for safety. And this feeds back into The Man Who Sold The Moon when this reactor in space blows up.

In these early stories we can already see that Heinlein has a complex view of society. In Life-Line and Let There Be Light corporate power is the villain of the story, and some of this also shows up in Blowups Happen. But in The Man Who Sold The Moon we see that private enterprise is preferred to government action. I think the way this can be reconciled is to see that Heinlein is always concerned with individual personal freedom and opposed to anything that might endanger that, whether from too much government or too powerful corporate interests.

The Green Hills of Earth contains the story of the same name, which concerns a former space engineer, Rhysling, now blinded by radiation and unemployable, who is also a poet. And one of his poems has that title. The crew of Apollo 15 named a crater on the moon “Rhysling”, and they planned to read a bit of it at the crater, but those trips could get very busy. Still, as they were getting ready to leave the moon there was this exchange. Note that Allen is the Capcom, and Scott and Jones are the astronauts :

“Allen: As the space poet Rhysling (the blind poet in Robert Heinlein’s The Green Hills of Earth) would say, we’re ready for you to “come back again to the homes of men on the cool green hills of Earth.”

[Scott – “That’s from the Green Hills of Earth. That’s one we talked about before the flight. Have you read that one?”] [Jones – “Oh, yeah! That was a favorite when I was a kid. Had you read it?”]

[Scott – “Sure. (Quoting from memory):

We pray for one last landing
on the globe that gave us birth
To rest our eyes on fleecy skies
and the cool green hills of Earth.”

Although two of the stories in this collection were older, from 1941, most of them are from 1948 and 1949. And there is a reason for that. On December 7, 1941, the United States found itself at war with Imperial Japan, and few days later Nazi Germany. Coming from a family that had fought in every American war you would expect Heinlein to get involved somehow. He could not enlist due to his medical retirement from the Navy, but since he had an engineering background so he became a civilian employee at the Philadelphia Navy Yard, where he was joined by fellow science fiction writers Isaac Asimov and L. Sprague de Camp. A nice retelling of this can be found at Kirkus Reviews, and Asimov also discusses this in his biography. The upshot is that there is a gap of about 5 years when Heinlein did not publish anything.

It is also notable that Heinlein by this point had escaped from the pulp science fiction magazines and gotten published in what were called the “slicks’, so-called because the paper they were printed on was slick and higher quality than the pulps. His stories began to be published in places like The Saturday Evening Post, Argosy Magazine, and Town & Country. And these outlets paid higher rates than the pulps, a significant matter for any writer. Heinlein always maintained that the only reason anyone would write was to make money.

And the stories were getting to be quite good as well. Delilah and the Space Rigger (1949) tells the story of a woman who joins a construction crew on a space station and faces discrimination, but wins out in the end, which was pretty progressive for the time, but not atypical for Heinlein. Space Jockey is a fairly pedestrian story about a rocket pilot dealing with his every day life. But The Long Watch is an important story to Heinlein’s view of the important things in life. A young officer is assigned to duty on the lunar base, where there are nuclear weapons stored. His superiors want to stage a coup, using those weapons, which can threaten the Earth while being beyond the reach of retaliation. The young officer sacrifices himself to prevent their plot from succeeding, and becomes recognized in a death as a great hero. And this becomes part of the background to a later juvenile novel Space Cadet, as well as being referenced occasionally in other stories, so you can see that he regarded it as an important statement. Gentlemen, Be Seated is a cute little story about a man who saves people when a leak happens in a tunnel on the Moon by plugging the leak with his rear end. The Black Pits of Luna is little thing about a boy scout who is able to rescue his little brother, but it foreshadows the Juvenile novels he later wrote.

It’s Great To Be Back! is about a couple who have moved to the Moon, but continually find fault with the living arrangements. They finally decide to go back to Earth, but discover that it was not really the place they had remembered, and they then return to the Moon, which they now realize is home. -We Also Walk Dogs is a gem of a story concerning a company called General Services that basically does things for their clients. Their advertising slogan is “Want somebody murdered? Then DON’T call General Services. But for anything else, call…. It Pays!” They deal a few different problems in this story, but the main one is the development of anti-gravity, and it features a Chinese porcelain bowl. Ordeal in Space is about a spaceman who has an accident that gives him a fear of heights and washed him out of space. But he has to face his fear when he needs to rescue a kitten from the 35th floor. One thing about Heinlein is that he was a firm and devoted cat fancier, so it no accident that a kitten is the one that has to be rescued. And the final story, Logic of Empire, he discusses the development of slavery in the Venus colony as a natural consequence of machinery being expensive and humans being cheap. And in this story there is a background reference to Nehemiah Scudder, who will soon be important in the Future History.

One of the things that is worthy of a brief discussion at this point is exemplified by the story Logic of Empire, and that is the reference to the Venus colony. We now know that Venus can best be described as hellish, with crushing air pressure and temperatures high enough to melt metals. The best designed landers can last no more than minutes before being destroyed. But this was not known when Heinlein was writing these early stories. The prevailing view at that time was that Venus was shrouded in clouds because it was very wet and swampy, so that is what Heinlein went with. Similarly his Mars had canals and was inhabited. You just have to go with it in these stories, as you have to do with so much of Golden Age science Fiction, let alone pre-Golden Age.

Links:

• https://www.facebook.com/HeinleinSociety/posts/i-never-created-or-invented-a-future-history-on-april-fools-day-1939-i-started-t/1092968002874634/
• https://www.amazon.com/Man-Who-Sold-Moon/dp/0671578634
• https://www.amazon.com/Green-Hills-Earth-Robert-Heinlein/dp/0671578537
• https://www.amazon.com/dp/B0011GBTKM/
• https://www.kirkusreviews.com/news-and-features/articles/asimov-de-camp-and-heinlein-naval-aviation-experim/
• https://www.palain.com/science-fiction/the-golden-age/robert-a-heinlein/the-future-history-part-1/

Provide feedback on this episode.

In this episode Kevie does a step-by-step approach to record an episode of HPR using the FFMPEG tool on the Linux command line.

Before beginning please ensure that FFMPEG is installed, it is available in the vast majority of Linux repositories.

Start by making a new folder to keep all your files in and move into the new folder (these will be numerous by the end of your recording):

mkdir Podcast
cd Podcast

To start recording audio use the command:

ffmpeg -f pulse -i default file01.flac

and finish the recording by pressing ctrl+c. I would recommend recording a test piece of audio to ensure that you are recording from your desired microphone and that the levels are to your liking.

To listen to the audio file we use ffplay:

ffplay file01.flac

Once all of the files have been recorded, to reduce the need for editing I would recommend recording several short segments, we need to put these together using:

ffmpeg -i file01.flac -i file02.flac -i file03.flac -i file04.flac -i file05.flac -filter_complex "[0:a][1:a][2:a][3:a][4:a]concat=n=5:v=0:a=1" filedone.flac 

Not that the number of sets squared brackets [] should be the same as the number of files (these start at zero) and the number after n= should be the actual number of files you wish to combine.

To remove any extended periods of silence then we can use:

ffmpeg -i filedone.flac -af silenceremove=stop_periods=-1:stop_duration=1:stop_threshold=-45dB filefinished.flac

Please note that this is a bit flaky at the time of recording (my results have been mixed) and it will re-encode the audio file so never do this with a lossy file such as ogg or mp3 as this will reduce the quality, keep this for lossless versions such as flac or wav files.

If you want to spend a bit more time editing the files and getting a better final audio file then the most effective way (but not quick) is to trim the audio from the end and beginning. Listen to the audio files and note the times of any periods of silence. As these normally are at the beginning and end, especially when we are recording in short segments then I will limit the instructions to avoid these becoming silly in length.

Firstly clip off the end silences first, if you start with the beginning then it will change the starting position of the end silences. To remove audio from the end of a track use:

ffmpeg -i file01.flac -vn -acodec copy -to 00:01:30 file01cut.flac

In this example anything after 1 minute and 30 seconds will be removed. The edited audio file will then be saved as file01cut.flac. This method does not re-encode the audio so there is no loss of quality.

To remove audio from the start of a track use:

ffmpeg -ss 30 -i file01.flac -c copy file01cut.flac

In the above example the first 30 seconds of the file will be removed and saved as file01cut.flac

Once you have edited each audio file then they will need to be merged together again to make a complete show.

Provide feedback on this episode.

01 Introduction

This is a follow up to my 4 part series on simple podcasting.

In this episode I will discuss a number of experiments with audio filtering.

These experiments were inspired by comments by listeners and by other discussions about audio on HPR.

I am not an audio expert, so I am doing this partly in order to learn something, but mainly in order to have a bit of fun.

I hope that you find this entertaining as well.

In a comment on the first episode a listener mentioned something called Solocast and said that the method bore a resemblance to the method that I was using.

Here is his comment

--------------------

02

Comment #3 posted on 2026-04-03 07:49:58 by Reto

It reminds me about Solocast

Hi Whiskeyjack,

I really liked your podcast and the topic. I cannot remember about your last, but the sound quality of this one was good on my mobile speakers :)

The concept reminded me about the program from Norrist (another host on HPR), while similar does it have some differences HPR 3496 https://hackerpublicradio.org/eps.php?id=3496

As I am not on the future feed, I look forward to your next episode.

Cheers, Reto

--------------------

03

End of comment.

I did not recall having heard the episode on Solocast, but this sounded very interesting. Solocast was in HPR episode 3496 and was released by norrist on the 27th of December 2021.

I listened to that episode and does indeed use use the same basic concept of recording short segments of audio and combining them later instead of creating one big recording and editing it with an audio editor.

04

The main difference is that the work flow that I described involves a lot of manual steps, while Solocast is a short Python program that automates the entire process of presenting your script, recording the segments, combining the segments, and filtering and normalizing the result.

I won't try to describe Solocast in detail, instead I would recommend just listening to HPR episode 3496 to get norrist's explanation directly.

--------------------

05

While I wanted to make sure that I credited norrist with having come up with this concept four years before I did, this won't be the focus of this episode.

Instead I will talk about audio filtering and various experiments that I ran on several different methods.

06

While looking at the source code for Solocast I noticed that it used a filtering method that resembled one used by Jivetalk, a podcast production program that caught the attention of one of the HPR community news presenters.

This method involves taking a sample of quiet audio where there is no speaking taking place, and then using this as input to a noise reduction filter which is applied to the voice recording.

The filter subtracts the quiet sample from the voice audio, which should theoretically remove the ambient noise.

07

I decided to apply this method to a number of different audio test recordings which were recorded under different circumstances using different hardware.

In this way I could see if the method worked equally well under all circumstances or if there were some sorts of noise which it was suited to and some sorts that were not.

08

While I was at it, I also picked several other filter methods to see how they worked as well.

Potentially, some methods may be better under some conditions while other methods were better suited to others.

--------------------

09

I won't present all of my experiments, as that would be a bit dull to listen to.

Instead I will describe each method and then present audio samples which illustrate my conclusions.

There are two pieces of audio software involved, both of which were also used in my series on simple podcasting.

10

The first is Sox, spelled s o x , and which is short for Sound Exchange.

Sox is a command line program for audio manipulation.

Sox is Free Software, released under the GPLv2 or later.

The other is FFMPEG, which is also a command line program.

FFMPEG is also Free Software, released under the LGPL V 2.1 or later, and GPL v 2 or later.

Sox actually uses FFMPEG for certain operations.

--------------------

11 Audio Hardware

For recording hardware I used the following.

12 Maxwell Headset

The first is a cheap Maxwell headset that has an electrical noise problem. Unfortunately I don't have a model number for this headset.

I described this hardware, the noise problems that I had with it, and how I created filters to deal with the noise in my series on simple podcasting.

Briefly though, this is a headset that has a build in microphone on a boom which allows the microphone to be positioned close to the mouth.

It connects with a USB cable.

13 Borne Earpiece and In-line Microphone

This is a set of earplugs that go in your ears and connected by wires and a very small microphone built into a small bulge in the cable. It connects using a 3.5mm jack.

The model number seems to be BUD250-BL.

14 XTrike Headset

This is a gaming headset similar to the Maxwell headset described above.

The model number is GH-510

It uses a USB connection.

15 Yanmai Condenser Microphone

This is a microphone that comes with a small tripod stand.

The model number is SF-910

It uses a 3.5mm audio jack.

--------------------

16

This is not a review of the hardware.

Rather, I was trying to create audio problems so that I could test ways to fix them.

Therefore, do not take the above list as a recommendation of what to buy.

However, you can see that I am not using any expensive audio hardware.

If you want to make an HPR podcast, you do not need professional level hardware.

--------------------

17 Audio Samples

The audio samples are as follows

18 Quiet

This was recorded in a quiet environment at my desk.

This is my normal podcasting environment and represents optimal conditions.

The main reason for this method is to see how the various filter methods perform when dealing with the electrical noise from the Maxwell headset.

19 Small fan

This is a small USB powered table fan approximately 10 cm in diameter.

It was located roughly 40 cm or less to the left of the microphone, although this varies depending on the microphone.

20 Traffic

This was along a busy street with traffic noise in the background.

--------------------

21 Filter Methods

Sox noisered Filter with Audio Profile

This method uses the Sox noisered filter. Here is a brief quote from the Sox documentation on this filter.

Quote

Reduce noise in the audio signal by profiling and filtering. This effect is moderately effective at removing consistent background noise such as hiss or hum. To use it, first run SoX with the noiseprof effect on a section of audio that ideally would contain silence but in fact contains noise - such sections are typically found at the beginning or the end of a recording.

End of quote

For these tests I recorded a separate noise profile to go with each test.

--------------------

22 Basic Manual Filter

This is a basic high and low pass filter pair based on the work I had done in my previous series on simple podcasting.

However, based on the tests that I have done for this episode, I decided to get a bit more aggressive in terms of filtering.

I use a high pass filter of 120 Hz, and low pass filter of 8 kHz.

The each filter is then applied twice to increase its effect.

I also added band reject filters to deal specifically with 50 and 60 Hz line noise.

--------------------

23 Complex Manual Filter

This uses the manually constructed filter described in my series on simple podcasting.

This uses the basic manual filter plus a series of custom bandreject filters to fix specific noise problems with the Maxwell headset.

--------------------

24 FFMPEG afftdn Filter

The documentation describes this as

"Denoise audio samples with FFT."

--------------------

25 FFMPEG arnndn Filter

The documentation describes this as

"Reduce noise from speech using Recurrent Neural Networks."

--------------------

26 FFMPEG agate Filter

I will pronounce this as "agate" for convenience.

The documentation describes this as

"A gate is mainly used to reduce lower parts of a signal.

This kind of signal processing reduces disturbing noise between useful signals."

--------------------

27 Method

The experimental method used was to take each noise sample and apply the different filter methods to it.

Where there are parameters which can be adjusted, a script was used to generate a series of different sample files with different parameter values.

Not all possible parameters were experimented with, as the goal is to see which method produces what sorts of results under different circumstances, not to get the best possible result for the samples that I happen to have.

The method in each case was as follows

28 Step 1

Convert the audio file to FLAC if it is not already in that format.

29 Step 2

Apply a basic high and low pass filter described previously to each sample.

The reason for this basic filtering is that it eliminates at least some undesired noise in a fairly fool proof manner, leaving less for the more advanced filter to deal with.

This should allow for a better test of the filter under realistic conditions.

30 Step 3

Apply the noise reduction filter being tested.

31 Step 4

Normalize the filtered sample to 17 LUFS according to the EBU R128 standard.

The EBU standard is described in my series on simple podcasting.

Normalizing adjusts the audio signal to a desired loudness level.

This allows for more more consistent sound levels and allows us to hear the results under realistic conditions.

I normalize the audio individually for each sample as different recording hardware requires different amounts of loudness adjustment.

This is different from the typical podcast process where normalizing takes place as the very last step in the process, but it was necessary in this case.

32 Step 5

Concatenate selected sample audio files to one another to allow for better review and comparing.

--------------------

33 Results

The results are grouped according to the type of noise which is being mitigated. This allows for easier comparison of the effectiveness of each technique under different circumstances.

I have only picked a few examples of interest out of the numerous experiments that I conducted.

--------------------

34 Quiet Recording Environment with Maxwell Headset

This compares how well the various filtering methods work on the noise induced by the electronics in the Maxwell headset.

This electronic noise consisted of a noise spike every 1 kHz.

This should be representative of electronic noise caused by problems in recording hardware.

35 Manual Filter

The manual filter applied a narrow band reject filter every 1 kHz from 1 kHz to 12 kHz.

This completely removed the otherwise audible whine caused by the noise.

36 FFMPEG afftdn

This method allows for setting a noise floor and then specifying how much the noise floor should be reduced by.

The method is very sensitive to getting the noise floor correct for that recording. Set the floor too low and nothing happens. Set it too high, and some distortion results.

However it seemed to be moderately effective, but it would seem to require checking it and possibly adjusting it each time it is used.

37 FFMPEG agate

This method allows setting a noise floor and then suppressing all sound which falls below that level.

This method is very sensitive to getting the noise floor correct for that recording.

If set too low (or quiet), it is ineffective.

If set too high (or loud), it distorts words which come after a pause, which would typically be between sentences.

38

When set correctly, it completely removes noise in the silences between sentences.

However, the noise is still audible during speech.

This is because the noise in this case is a higher frequency than normal speech, and so stands out more.

It may not be a significant problem for noise which is closer to the main vocal frequency band.

Overall, this method is not suitable for this particular problem.

39 FFMPEG arnndn

This method used the standard model.

A variety of different noise reduction models are available.

I only tested it with one, std.rnnn

It does not seem to introduce much distortion in the voice signal even with a high amount of mix parameter.

40

However, it is only slightly effective at removing the whine from the signal, even with a high amount of mix parameter.

Overall, this method does not appear to be useful for this sort of noise problem.

41 Sox noisered Filter

This was effective in removing noise between words, but noise can be heard while words are being spoken. It was better than agate however.

42 Overall Conclusion for the Maxwell Headset Noise

When dealing with narrow noise bands that occur at known frequencies, the manual filter is leagues ahead of any of the other tested alternatives.

43 Sample Audio

Here is a sample audio recording showing the best overall results

The sample is repeated, first with only basic low and high pass filtering, and then with the manually constructed filtering.

In the first sample you should hear a high pitched background whine.

In the second sample, the high pitched whine is completely removed.

44 (Audio sample inserted here.)

--------------------

45 Traffic Noise

This was recorded using the Borne in-line microphone connected to a mobile phone while walking along beside a busy street.

This was in dry cool spring weather, and the road was paved with asphalt.

This should be reasonably representative of podcasting while walking outdoors in a noisy environment.

46 Basic Manual Filter

This used the basic manual filter with high and low pass filters. This did nothing very useful in this case as the signal was already filtered within those limits by the recording hardware anyway. The low sample rate of 8 kHz in the phone limited the upper frequency to 4 kHz. Recall that the sample rate has to be twice the highest frequency that you want to detect.

Overall, this is not suitable for this sort of problem.

47 FFMPEG afftdn

With a high noise floor, background noise is reduced, but not eliminated. There was not much distortion in the voice.

This is only slightly useful for this sort of problem.

48 FFMPEG agate

With a high threshhold, background noise is reduced, but not eliminated. There was some distortion in the voice.

The background noise could also be heard when speaking, but because the frequency of the background signal was similar to the louder voice signal, it was not as noticeable as it would have been if the two were very different.

This is moderately useful for this sort of problem. It may be more useful in situations where the background noise was not quite as loud.

49 FFMPEG arnndn

With high amounts of noise reduction, much of the background noise is suppressed, but there is not a lot of distortion in the voice.

The background traffic noise is still present, but is significantly less.

This offers only a moderate improvement.

50 Sox noisered Filter

With small amounts of noise reduction voice is clear but traffic noise is present as a very significant continuous warbling sound in the background.

This is no improvement on the original and in fact could be seen as making it worse.

With moderate amounts of noise reduction, traffic noise is mostly gone, but there are still various squeaks present. Voice is noticeably distorted.

With large amounts of noise reduction, traffic noise is gone but voice is highly distorted.

This is moderately useful for this sort of problem, but requires careful adjustment.

51 FFMPEG arnndn Followed by FFMPEG agate

This combined two different filters.

First, it used arnndn to suppress the background noise to a lower level without much voice distortion.

Then it applied the agate filter to suppress the noise levels between words still further.

This used the same amount of mix and threshold as was found to be most effective when each of these filters was used on its own.

The background noise is almost completely gone while distortion of the voice signal is low.

52 Overall Conclusion for Traffic Noise

The arnndn combined with agate filters was the most successful at suppressing background noise while limiting the amount of voice signal distortion.

53 Sample Audio

Here is an audio sample for what I felt to be the best overall results, the arnndn filter combined with the agate filter.

First is the original audio with basic filtering.

This is followed with the same audio after being passed through the arnndn and agate filters.

54 (Insert arnndn plus agate audio sample here)

55 Another Sample

Here is a second audio sample showing the Sox noisered profile based filter.

I have included this to show how a profile based filter can make things worse if you are not careful how you use it.

This repeats the test audio 4 times.

The first is with basic filtering only.

The second uses low amounts of noise reduction.

The third uses moderate amounts of noise reduction.

The fourth uses high amounts of noise reduction.

56 (Insert noisered audio sample here)

--------------------

57 Small Fan Noise with Yanmai Microphone

This was recorded using the Yanmai condenser microphone.

A small fan was set up behind and to the left of the microphone.

This is intended to represent situations where someone may have a fan or air conditioner running in the background due to hot weather, or has a loud computer fan.

58

A condenser microphone was used for this test as they are more prone to picking up unwanted noise.

However, for practical recording purposes, this sort of microphone is unsuitable for this type of environment.

59 Basic Manual Filter

This used the basic manual filter with high and low pass filters.

This did nothing useful as the fan noise was in the same frequency range as the voice signal.

This may be of more help in cases where the noise is below the 120 Hz cut off used in the low pass filter.

60 FFMPEG afftdn

With high amounts of noise reduction, much of the background noise is suppressed, but there is some distortion in the voice.

The background fan noise is still present, but is significantly less.

Overall this is moderately effective.

61 FFMPEG agate

This was effective in removing noise between words, but noise can be heard while words are being spoken.

However, this was a small voice sample and it is possible that more problems could occur.

With less fan noise than was in this sample this technique may work much better.

62 FFMPEG arnndn

With high amounts of noise reduction, much of the background noise is suppressed, but there is not a lot of distortion in the voice.

The background fan noise is still present, but is significantly less.

Overall this was fairly effective.

63 Sox noisered Filter

With small amounts of noise reduction voice is clear but fan noise is present as a slight warbling sound in the background.

With moderate amounts of noise reduction, fan noise is gone, but voice is somewhat distorted.

With large amounts of noise reduction, fan noise is gone but voice is very distorted.

64

In general this method is fairly successful at dealing with this sort of problem.

However, there is a trade off between background noise and voice quality.

Getting that trade off correct takes experiment and judgment for each specific situation.

65 FFMPEG arnndn Followed by FFMPEG agate

This combined two different filters.

First, it used arnndn to suppress the background noise to a lower level without much voice distortion.

Then it applied the agate filter to suppress the noise levels between words still further.

This got rid of virtually all of the background noise between words.

If you listen carefully however, there is a slight buzzing sound in the voice signal.

66 Overall Conclusion for Fan Noise with Yanmai Microphone.

Of the methods tested, the arnndn followed by agate filter seemed to offer the most improvement for the least effort and least voice distortion.

The arnndn filter on its own seemed the next most preferable to me despite leaving some fan noise in the background.

67 Audio Sample

Here is an audio sample for what I felt to be the best overall results, the arnndn filter combined with the agate filter.

First is the original audio with basic filtering.

This is followed with the same audio after being passed through the arnndn and agate filters.

68 (Insert audio sample here)

--------------------

69 Small Fan Noise Recorded with Headset

The following is an observation rather than a filtering technique.

When a recording was made using the Maxwell headset and listened to on the headset later or with speakers, the fan was virtually inaudible.

When the same recording was listened to with the XTrike headset, it was barely audible with careful listening and only identifiable as a fan because I knew it was there.

70

In situations where there is ambient noise, the best noise reduction technique is probably to move the microphone as close to your mouth as possible, although not directly in front of it, and reduce the gain if there is a gain adjustment in the microphone.

This will work far better than trying to remove the noise later.

If you are recording an HPR episode at a desk, then an inexpensive headset with boom mike may do the job just fine with minimal effort and expense.

--------------------

71 Conclusions

I have tested three noise scenarios -

Electronic noise in the audio hardware at specific frequencies.

Recording outdoors with an inline microphone in a noisy traffic environment.

A noisy fan creating background noise in an office.

My conclusions on these are as follows.

72 Electronic Noise in the Audio Hardware at Specific Frequencies

If you can use Audacity or some other means to find the frequencies which are causing the noise, the best solution, assuming you don't just replace the hardware, is to manually construct filters to remove those specific frequencies.

This is the safest solution in terms of only doing what you tell it to and not producing unexpected surprises some time down the road when something changed in the environment.

73

If you are looking for a fairly automatic filtering method, the Sox noisered profile based filter seems to work fairly well.

There is an equivalent filter in ffmpeg, but I did not include that in my experiments as it is harder to use in a script because it does not use a separate noise profile file.

74 Recording Outdoors with an Inline Microphone in a Noisy Traffic Environment.

In this situation, the FFMPEG arnndn combined with agate filters seem to be the most successful.

The Sox noisered filter may work, but at the cost of more distortion in the voice than is seen in the other methods.

75

An inherent problem with any profile based noise reduction method is that if the background noise is not constant, which it seldom is in that sort of environment, the profile may not represent the background noise which is present later on in the recording.

This risks adding more distortion in the voice as the profile and later environments diverge.

76

However, for this application a different microphone that provided a better recording would appear to be advisable.

A solution which brought the microphone much closer to the mouth and so resulted in a better ratio of voice signal compared to background noise would appear to be necessary, after which the question of what sort of noise reduction to use would need to be re-evaluated.

77 A Noisy Fan Creating Background Noise in an Office.

The Sox noisered filter and the FFMPEG arnndn, afftdn, and agate methods all work to some degree.

However, they all need correct selection of parameters to achieve the proper results.

When I compared all four methods side by side, I found the arnndn combined with the agate filter to be preferable in terms of the trade off between background noise reduction and distortion of the voice signal.

The arnndn filter on its own seemed the next most preferable to me despite leaving some fan noise in the background.

78

However, that is a subjective judgment of a specific noise sample when recorded using a specific microphone.

Keep in mind though that many listeners will not be listening in an idea environment.

They may be doing things where background noise is present rather than in a very quiet room and so may find a small amount of background noise in the recording to be less of a problem than distortion in the voice signal which may make some words harder to understand.

79

When I conducted the same experiment recorded with the XTrike headset I found that arnndn seemed to offer no noticeable improvement.

This may be because the amount of audible fan noise was far less with the XTrike headset to begin with.

In other words, there is no single best solution here, and you may have to be prepared to try different options to see which one works in your situation.

The important thing is to avoid making things worse by applying filtering that is not appropriate for that situation.

The best method may be to use a recording method that doesn't pick up the fan noise to begin with.

This can include just using a gaming headset with boom mic.

80

I have one final observation on this point regarding headsets.

The Maxwell headset has a foam cover over the microphone while the XTrike headset does not.

There was some slight audible wind buffeting noise picked up by the XTrike headset that was not observed with the Maxwell.

This seemed to cause particular problems with the Sox noisered profile based filter, as this noise was irregular and after filtering would show up as a warbling sound.

If you use a headset and plan to use it in conjunction with a fan, it may be advisable to apply some sort of wind cover over it.

81 Combining Complex Filters

In several cases I found that combining several complex filters offered better results than using any single one on its own.

The basic strategy though is to first use a method which is good at reducing undesirable noise without introducing excessive voice distortion.

Then apply a different filter which is good at reducing small levels of background noise to an even lower level while affecting the voice signal as little as possible.

This uses the relative strengths of different filter types to compensate for the weaknesses of the other.

82

Different combinations of filters were most effective for different types of problems.

I did not try all possible combinations however.

Perhaps a further exploration of this would be worth doing in a later podcast.

--------------------

83 Case Study - Noise in Another HPR Episode Audio

In the comments to my second episode on Simple Podcasting (which is HPR4618) where I discussed basic filtering, a couple of listeners brought up an interesting point.

Antoine mentioned "declicking" in a post.

--------------------

Vance replied

84

Antoine, thanks for mentioning the click removal capability in Audacity! While I already knew about its noise removal filter, I wasn't aware it also had click removal.

It might have helped me for HPR4637, where some sort of electromagnetic signal was picked up by my microphone/recorder, a Zoom H2 (the tapping sound was *not* present in the room where I recorded).

While click removal does seem to distort speech when applied to it (though to my ears, it doesn't sound as weird as when noise removal is done with speech), I could have applied the filter only to the pauses, where the "tapping" is most noticeable. I will consider doing this in the event that I'm not able to eliminate the source of interference in the future, which would be the best way to go.

--------------------

85

End of quote.

I found this interesting as it sounded like another audio problem that could be experimented with.

I found a sample of the episode which had the clicks and cut a copy of that segment out to experiment with.

These sounds are a series of clicks, or "ticks" would be another way to describe them, in the quiet part of the audio between sentences or phrases.

86

Next I used Audacity to study the sound spectrum.

I found a massive 60 Hz noise spike.

However, my speakers won't reproduce sound that low, and filtering this out didn't reduce the clicks.

The clicks turned out to be bursts of noise across the 100 to 800 Hz band, which is right where the main vocal band also is.

This makes it difficult to filter based on frequency.

The most promising approach would seem to be to filter based on sound level.

87

I tried all of the individual audio filter techniques mentioned in the other experiments above.

None produced satisfactory results except for agate, which makes quiet audio quieter.

This completely suppressed the clicks.

However, when applied to the entire episode it also distorted the start of a few sentences which began with single short syllables.

88

The agate filter has a number of parameters which could be adjusted to try to deal with these cases, although I did not spend the time to do so.

Another solution to this distortion problem is to simply not apply the filter to those parts of the audio which are affected.

If you record the audio as a series of small individual files, it would be easy enough to filter before concatenating the files together while skipping those files which contain audio which is not suited to this method.

Here are the results of the experiments.

89 FFMPEG afftdn

This reduces the size of of the ticks, but they are still present.

However, they may be reduced to a level which is considered acceptable.

90 FFMPEG agate

This was very effective in removing ticks with the right parameters.

However, it can introduce some voice distortion in the form of cutting out the start of a few sentences which began with single short syllables.

This can be corrected with a very short "attack" parameter to turn off the filter when it detects sound above a set threshhold.

91 FFMPEG arnndn

This was relatively ineffective.

92 Sox noisered

This was effective in removing the sounds between phrases.

However, it introduces some distortion in the voice signal.

93

I also tried combining filters.

FFMPEG afftdn Followed by agate

This combined two different filters.

First, it used afftdn to suppress the background noise to a lower level without much voice distortion.

Then it applied the agate filter to suppress the noise levels between words still further.

This got rid of virtually all of the background noise between words.

94

Here is a short audio sample from HPR4637.

First is the unfiltered audio.

Second is the filtered audio using the combined afftdn plus agate filters.

Since the "clicks" are very quiet, you may not hear them unless you are in quiet environment.

Quite a few listeners would probably not be aware of the perceived audio problem in this episode if it had not been discussed here.

None the less, it makes for an interesting experiment.

Here it is:

95 (Insert sample audio here)

96 Overall Conclusion for Noise "Ticks"

The afftdn combined with agate filters seemed to offer the best overall results when used with the right parameters.

However, the author, Vance, speaks very clearly and evenly, and so his voice is ideally suited for use with this filter.

Another author's voice may not be as suited to this filter.

97

The Sox noisered profile based filter offers various degrees of trade off between suppressing noise and distorting the voice signal.

As to whether this is an acceptable trade off depends on the particular voice in question and how easily understood it is under normal circumstances with out additional distortion.

The afftdn filter may be a fairly safe filter to use on its own while producing acceptable if not perfect output.

--------------------

98 Overall Conclusions

I have presented only a few of the experiments that I conducted.

My overall conclusion after all of this is that there is no universal audio filtering method that works best in all circumstances.

There are instead a number of tools in the toolbox, and picking the right one for the job takes a bit of trial and error.

99

However, if you have a repeatable recording environment, then once you have decided what tool you need you should create a script for it so you can have a repeatable processing setup.

These conclusions apply to voice podcasting.

Music has a different set of criteria and techniques that work well with basic voice podcasting may produce poor results when applied to music which has a broader range of frequency and just as importantly, a broad range of loudness.

100

If you are used to using filters and effects in Audacity, many of the settings on those correspond to arguments in the command line version of ffmpeg.

It is worth learning how to use ffmpeg directly to automate your recording process.

101

The experiments that I conducted were greatly assisted by writing scripts which created multiple versions of audio files with different settings, thereby allowing me to try many different alternatives relatively easily.

It also allowed me to concatenate different audio samples into a single audio file and so listen to different versions in quick succession, making subjective listening judgments more reliable.

102

It is important to keep in mind in all this that I am playing with audio filtering mainly to have fun.

It is not necessary to do any of this if you think your podcast episode sounds just fine without it.

So, don't let any of what I have talked about in all this discourage you from simply recording a podcast and sending it in as is.

I will include copies of the filters I have described here in the show notes.

--------------------

103 Related Matters

Hardware Characterization Using Audio Signals

I found it useful to characterize the hardware that I had in order to understand its limitations better before starting the experiments.

This involved playing a signal out through a set of speakers and then recording it through a microphone.

104

I used two types of signal for this.

One is type of signal is known as a "chirp" signal.

This is a sine wave that steadily increases in frequency as it sweeps across the audio spectrum.

The standard audio range is 20 Hz to 20 kHz, but for my purposes I limited the upper frequency to 15 kHz to save time as anything beyond that is not very useful for voice podcasts.

105

By recording the chirp signal with a microphone and analyzing it with a Fourier transform, I could quickly see what each device was capable of.

See my previous series on simple podcasting for an explanation of what a Fourier transform is and what software to use to see the results of it.

Here is a chirp signal.

106 (Insert Audio Sample Here)

107

In addition to a chirp signal, I also used a series of simple tones of specific frequencies.

By using these tones of known frequency I could gain an understanding of the limitations of my speakers and headphones, and just as importantly, my own ears.

By understanding these limitations I was able to narrow the range of frequencies that I need to deal with quite considerably and set the high and low pass filters accordingly.

These tones are a series of flac files generated with ffmpeg.

108

Here is a a sample audio tone at a 2 kHz frequency.

109 (Insert Audio Sample Here)

110

Copies of the script to create the chirp signal and the tones are in the show notes.

--------------------

111 A "Not a Review" of some of the Hardware that I Used

I said that I would not do a review of the hardware that I used.

However, some of it deserves mention for either how good or bad it was.

I will record each section using the hardware being described.

112 Maxwell Headset

This is my original recording hardware.

This is a headset with boom mic and USB connection.

There is no model number on it, so I don't know the model.

This probably cost somewhere between 10 and 25 dollars.

The earpieces sit on the ears and do not fully enclose them.

This makes it light weight and comfortable to wear for extended periods of time.

It has a problem however with electronic noise consisting of a noise spike every 1 kHz.

I was able to fix this with a series of filters using FFMPEG.

Fixing this problem is what got me started in understanding audio.

I will probably continue to use this headset to make podcasts.

113 XTrike Headset, Model GH-510

This is also a headset with boom mic and USB connection.

I purchased this headset for the purposes of experimentation for this podcast episode.

It cost $12.88.

I found it to be surprisingly good for the price.

It has fully enclosed ear pieces however, which may make it uncomfortable to wear in hot weather.

I may try doing some of my future podcasting using this headset.

114 Borne Earpiece and In-line Microphone

This is a set of earplugs that go in your ears and connected by wires and a very small microphone built into a small bulge in the cable. It connects using a 3.5mm jack.

The model number seems to be BUD250-BL.

It cost approximately $3.00.

I bought several sets of these and use them for listening to podcasts from an MP3 player.

The ear pieces are pretty good for listening with.

The microphone works reasonably well when used in a quiet location.

It is less good when in a noisy environment.

It is very important however to secure the microphone to your lapel or other location reasonably near your mouth and to point the microphone (that is the small hole) outwards and not simply let it dangle freely.

If you let it just hang, you will get poor quality and inconsistent audio.

115 Yanmai Condenser Microphone, Model SF-910

I purchased this microphone for the purposes of experimentation for this podcast episode.

It cost $3.88.

As it is a condenser microphone, it is prone to picking up background noise more and as such is probably not a good choice for podcasting by single person sitting at a desk.

However, it is none the less a surprisingly good microphone for surprisingly little money.

116 iCan USB Microphone, Model M-306

I purchased this microphone for the purposes of experimentation for this podcast episode.

This has a USB connection.

This was also relatively inexpensive at $7.99, or roughly twice the price of the Yanmai microphone.

Unlike the Yanmai however, it is absolutely wretched.

There was such a high degree of distortion when recording through it that I found I could not use it in the fan experiments which I had bought it for.

I ended up buying the Yanmai microphone for that instead.

--------------------

117 Easy Effects Software

The techniques described so far all involve recording audio files and then processing them later to produce the desired result.

This is probably the simplest and most straightforward way of doing things if you are making a typical podcast.

However, there may be instances where you want to apply filtering or other effects on the "live" signal immediately and not after the fact.

118

There is audio software which can hook into your computer's audio system and do this with a live signal.

For Linux, there is a package called "Easy Effects".

This is Free Software and comes under a GPL V3 or later license.

I installed it from the Debian repository under Ubuntu 24.04.

119

You can create various filters and even chain them together to combine them.

I played with it a bit but do not know enough about it to discuss it seriously at this time.

However, I thought it would be worth mentioning for the sake of those who may wish to try it out themselves.

--------------------

120 Episode Conclusion

After having had some fun with audio and listening to other HPR members talk about audio, I thought I would have some more fun by playing with noise reduction filters.

I have no intention of becoming an audio professional, but by doing some experiments I learned a few things and had some fun doing it.

I hope that the rest of you found this interest as well.

I will see you all again later in another episode of Hacker Public Radio.

--------------------

Scripts

Basic Filter

This shows basic high and low pass filters ( 120 Hz and 8 kHz respectively)

and band reject filters for 50 and 60 Hz.

# The high and low pass filters.

hlpfil="highpass=f=120, highpass=f=120, lowpass=f=8000, lowpass=f=8000"

# Band reject filters filter for 60Hz and another for 50Hz.

linefil="bandreject=f=60:width_type=h:w=20, bandreject=f=50:width_type=h:w=20"

# Filter using ffmpeg.

ffmpeg -i inputfile.flac -af "$hlpfil, $linefil" outputname.flac

# ======================================================================

afftdn Filter

# noisefloor should be between 20 and 80.

noisefloor=$1

# Run the noise reduction.

ffmpeg -i testrec-filtered.flac -af "afftdn=nr=10:nf=-""$noisefloor" tmptestrec.flac

# ======================================================================

agate Filter

# threshold shoud be between 10 and 80.

threshold=$1

# Run the noise reduction.

ffmpeg -i testrec-filtered.flac -af "agate=threshold=-"$threshold"dB:range=-60dB" tmptestrec.flac

# ======================================================================

arnndn Filter

# mix should be between 0 and 1.

mix=$1

# Run the noise reduction.

ffmpeg -i testrec-filtered.flac -af 'arnndn=model=std.rnnn:mix='"$mix" tmptestrec.flac

# ======================================================================

sox noisered Filter

# Generate the noise profile from a sample of background noise.

sox silencefiltered.flac -n noiseprof noise.prof

# nramount shoudl be between 0 and 1

sox testrec-filtered.flac noiseout-testrec.flac noisered noise.prof "$nramount"

# ======================================================================

Manual Filter for Maxwell Headset Noise

# Create a series of band reject filters, from 1 kHz to 11 kHz.

ftemplate="bandreject=f=%s000:width_type=h:w=100"

kilospikefil=$( seq 1 11 | xargs printf "$ftemplate," )

# Using ffmpeg

ffmpeg -i testrec-filtered.flac -af "$kilospikefil" tmptestrec.flac

# ======================================================================

Create a "chirp" signal

# Start frequency.

f0=20

# End frequency.

f1=15000

# Duration of signal.

duration=10

ffmpeg -f lavfi -i "aevalsrc=sin(2 * PI * (0.5 * ($f1 - $f0)/$duration * t^2 + ($f0 * t))):s=44100:d=$duration" -c:a flac -af "aformat=sample_fmts=s16" chirp.flac

# ======================================================================

Generate Audio Tones

toneout () {

printf -v freqval "%05d" $1

ffmpeg -f lavfi -i "sine=frequency=$freqval:duration=3" tmptone.flac

# Normalize

ffmpeg -i tmptone.flac -af loudnorm=I=-17:TP=-2.0:LRA=4.0 -ar 44.1k -sample_fmt s16 tone$freqval.flac

rm tmptone.flac

}

# List of frequencies in hertz.

freqlist="50 60 100 120 130 140 150 160 170 200 500 1000 2000 3000 4000 5000 6000 7000 8000 9000"

for freq in $( echo $freqlist ); do

toneout $freq

done

# ======================================================================

Provide feedback on this episode.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.

Most users of UNIX-like systems are probably familiar with the diff utility. It is widely used with source code to compare two files and see what the differences are between them. Non-programmers, like me, also use it to examine what has changed in different versions of scripts or configuration files. Quite a few pieces of newer software can compare different versions of data and express changes in a format either identical to or similar to diff output.

However, there are two other long-standing tools for this purpose that are far less known and deserve in my view to be termed UNIX Curios. The first of these is cmp ¹ . While diff is primarily intended to be used on text files and compares them line by line, cmp compares files byte by byte. In my experience, its main use is to see whether two binary files are in fact identical—if they are, cmp outputs nothing and returns an exit status of 0. Back when methods of transferring files were not as reliable as they are today, this was a tool I would reach for sometimes. For example, you could use it to confirm that the data on a CD-ROM you burned was the same as the original.

If there is a difference between the files, cmp will return an exit status of 1. By default, it will also print the location (byte and line number) of the first differing byte. When used with the -l option, it will print the location and value of every byte that differs. There is one exception to these: if the files are the same except that one is shorter than the other, it will print a message to that effect. The exit status will still be 1 in that case.

Using the -s option with cmp will cause it to be totally silent and output nothing. Only the exit status will indicate whether the files are the same, different, or if the exit status is greater than 1, that an error occurred. This makes it useful for scripting, for example in case you wanted to confirm that a file copied to another location arrived fully intact.

It is worth noting that diff is also capable of comparing binary files—however, it is not required by POSIX to report what is actually different or where differences occur. The same exit status as in cmp is returned: 0 if the files are the same, 1 if they are different, or greater than 1 if an error occurred. While many implementations offer an option to suppress the output, this is not in the standard ² so the most portable method would be to instead redirect output to /dev/null . On my system the diff utility is three times the size of cmp , so if you don't need its extra capabilities, it is a less efficient way of doing the job.

The other UNIX Curio for today is comm , and this utility ³ is also intended to compare two files to see what is common between them. Ken Fallon briefly talked about it a few years ago in HPR episode 3889 . Compared to the others, it has a much more specific use case. The two files are expected to be text files that are already sorted. What comm will do is print a tab-separated list of all the lines appearing in either or both files. Lines only in the first file will appear in the first column, lines only in the second file will be in the second column, and lines in both files will be in the third column.

Any combination of the options -1 , -2 , and -3 can be used with comm to suppress printing of the first, second, or third column respectively. Using all three options at the same time is supported but it results in no output, so that isn't very useful. Unlike the other utilities, the exit status of comm doesn't tell you anything about the two files. It will be 0 if the program ran successfully, and greater than 0 if it didn't.

I'm not sure if I have ever actually used comm for anything practical. I find its default output a bit difficult to meaningfully interpret, plus you need to ensure the two files are already sorted. It seems to be best suited to comparing lists, and one use case that Ken Fallon mentioned would be comparing two lists of files to see if any are missing. The command comm -3 listA listB would print files that only appear in listA in the first column and those only in listB in the second column. This would let you ignore all the filenames that appear in both and focus on those that were absent from one or the other. If on the other hand you only wanted to see the filenames that are on both lists, comm -12 listA listB would give you that.

Some more frivolous potential uses also come to mind. If for some reason the cat utility is broken on your system, you could use comm listA /dev/null to print the file listA instead. If you want to insert tab characters before every line of a file but have an aversion to using sed or awk , then comm /dev/null listA would output listA with one tab before each line, and comm listA listA would insert two tabs. A bit silly, but it would work. The GNU implementation of comm even lets you choose something other than a tab to separate the columns ⁴ , so you could go wild with that.

According to the POSIX specifications for cmp and comm , one of the two filenames given as arguments, but not both, can be a " - ", in which case standard input will be used for that "file" in the comparison. Also, the results are undefined if both arguments are the same FIFO special, character special, or block special file. Some implementations might not have these limitations, but you shouldn't rely on that everywhere.

All three of these were developed quite early. The cmp utility appeared in 1971's First Edition UNIX ⁵ , while comm and diff seem to have made their debut in Fourth Edition UNIX ^6,7 from 1973. The original versions might not have behaved exactly like their modern counterparts, and newer implementations (especially of the diff utility) have acquired additional options and capabilities, but the basic operation of each has stayed the same.

The next time you need to compare files against each other, consider whether cmp or comm might be appropriate before automatically reaching for diff . They all have their uses in different situations.

References:

1. Cmp specification https://pubs.opengroup.org/onlinepubs/009695399/utilities/cmp.html
2. Diff specification https://pubs.opengroup.org/onlinepubs/009695399/utilities/diff.html
3. Comm specification https://pubs.opengroup.org/onlinepubs/009695399/utilities/comm.html
4. GNU coreutils manual: comm https://www.gnu.org/software/coreutils/manual/html_node/comm-invocation.html
5. First Edition UNIX cmp manual page http://man.cat-v.org/unix-1st/1/cmp
6. Fourth Edition UNIX comm manual page https://www.tuhs.org/cgi-bin/utree.pl?file=V4/usr/man/man1/comm.1
7. Fourth Edition UNIX diff source https://www.tuhs.org/cgi-bin/utree.pl?file=V4/usr/source/s1/diff1.c

Provide feedback on this episode.

Sound quality, ease of use, comfort, battery life, price

Provide feedback on this episode.

HPR and Glasses

1. HPR community members are nerds
2. We know from movies that nerds wear glasses
3. We can conclude that HPR community members wear glasses

Cleaning Background

• I have very oily skin and am constantly smudging my glasses
• All lenses wipes are too small.
• I accidentally touch the lenses with my fingers when I am trying to clean them And smudge them up again
• I heard an eye doctor say, the best way to clean glasses is with dish soap and dry with a microfiber cloth
• I tried this method It is a great solution because you wash glasses and fingers tips at the same time

Cleaning Method

• Hand soap is OK to use on glasses.
• get the soap on you fingertips
• Then hold your glasses one of the lenses
• Use your thumb and finger tips to clean the glasses
• the soap removes the excess oil from your fingertips
• no smudges when you dry with the microfiber cloth

Progressives

• With regular progressives the top of the lenses is for distance vision and the bottom of the lenses is for up close vision
• There is a zone in the middle for medium distance, but it is very narrow
• Using a computer at work I kept having to move my head up and down to find the best position for working.
• Computer Glasses have more space for intermediate distances
• From https://iotlenses.com/blog/eight-types-of-progressive-lenses.html

Computer progressive lenses, also known as occupational designs or enhanced readers, are optimized for intermediate and near vision, making them ideal for tasks like prolonged computer work or activities at mid-range distance

Provide feedback on this episode.

Background

This episode describes the contents of an old fishing tackle box I got as a birthday present very many years ago. I used it over the years to gather a pile of mostly useless electronic components and junk.

I came across it recently when we were doing a clear out. Mrs X wanted to throw it out. I had a lot of sentimental value due to the long time I’ve had it. I am a bit of a hoarder and it took a bit of persuasion on my part to be able to keep it especially since I could not think of a single use for any of the contents!

I’ve not had time to add links against all the items. Hopefully I’ve managed to add links to most of the more unusual items. Apologies if I’ve missed any.

Contents of Box

Top row left to right, back to front

Compartment 1

Stereo Jack plug & socket

Double pole double throw (2-way) switch

12V dc relay from RS

3 legged semiconductor (voltage regulator?)

Compartment 2

Multiple fuses mostly for cars

Compartment 3

Small silica jelly pouch

Micro Switch

Reed switches (Not reed relays as I call them in my show)

Compartment 4

Potentiometers, Rheostat, variable resistors https://en.wikipedia.org/wiki/Potentiometer

Control knobs

Compartment 5

Large selection of miniature panel mounted toggle switches https://en.wikipedia.org/wiki/Switch#Toggle_switch

Compartment 6

Two crocodile clips https://en.wikipedia.org/wiki/Crocodile_clip

Half of a Hi Fi speaker plug https://en.wikipedia.org/wiki/File:Speaker_din_male_and_female.jpg

Washer

Compartment 7

Mostly chocolate block connectors https://en.wikipedia.org/wiki/File:Leuchterklemme_75dpi.jpg

Small selection of glass cartridge Fuses

Single 13 Amp mains fuse for UK plug

Assortment of soldering iron tips

Crocodile clip

Air adapter for foot pump for blowing up an air bed or similar

Spare nozzle for a de-soldering tool. Made from PTFE https://en.wikipedia.org/wiki/Desoldering#/media/File:Vacuum_plunge.jpg

https://en.wikipedia.org/wiki/Polytetrafluoroethylene

Ferrite ring https://en.wikipedia.org/wiki/Ferrite_bead

Compartment 8

Rubber Grommets and a heatsink

Middle row left to right, back to front

Compartment 1

Large selection of glass cartridge fuses

Compartment 2

Another large selection of glass cartridge fuses

Small selection of diodes, some quite Large https://en.wikipedia.org/wiki/Diode

Compartment 3

Two multi turn precision variable resistors https://en.wikipedia.org/wiki/Potentiometer#/media/File:Beckman_Helipot_potentiometer_SA1400A_2007.075.002.jpg

Cheap variable resistor with in built switch

Compartment 4

Bag of multi coloured LED’s (Cheap for Pi add on board)

2 inch 8 ohm speaker

Compartment 5

Panel mounted heavy duty toggle switch, two pole one way

Compartment 6

Light Dependent Resistor (LDR) https://en.wikipedia.org/wiki/Photoresistor

Large Tricolour LED

Compartment 7

Variable capacitors PCB (Printed Circuit board) mounted

Spare nozzle for de-soldering tool. Made from PTFE

Light Dependent Resistor (LDR)

Panel mounted heavy duty rotary switch, two pole one way

Bag of various fuses

Halogen 12V car headlight bulb

Variable resistor with long plastic spindle, panel mount

Heatsink possibly for heavy duty transistor

Compartment 8

Variable resistor PCB (Printed circuit board) mounted

Bottom row

This is just a big jumble of stuff, listed in no particular order

DC red and black power lead with inbuilt fuse

Small solder-less breadboard for prototyping

https://en.wikipedia.org/wiki/Breadboard

Bag of various plugs & connectors mostly Jack plugs and sockets

Bag of quartz crystals and a components receipt from 1985 I added crystals to an old PMR (Private mobile radio) boot mount radio. Possible a Pye Westminster Radio. I looked but couldn’t find picture of the radio

Several bits of Perfboard or strip board

https://en.wikipedia.org/wiki/Perfboard

1 Watt audio amplifier kit with instructions not built!

D-type serial port mounted to PCB with screw down connections

Bag of resistors https://en.wikipedia.org/wiki/Resistor

Large bag of LED panel mount holders

Electronic Instruction guide XH79L (Maplin) price 25p

Chocolate block connecting block https://en.wikipedia.org/wiki/File:Leuchterklemme_75dpi.jpg

Telephone pickup for recording phone calls. (Never worked) https://www.amazon.co.uk/Black-Telephone-Pickup-Sensitive-Microphone/dp/B000L1OZG8

Multi coloured ribbon cable

Plastic trimming tool screwdriver set. https://www.amazon.co.uk/C-K-T4857-Precision-Plastic-Trimmer/dp/B000WKIYTW

Panel mounted BNC socket (Baby Neill Constant)

https://en.wikipedia.org/wiki/BNC_connector

Provide feedback on this episode.

Openreads is a privacy oriented and open source cross-platform app written in Flutter available for Android and iOS for keeping track of your books.

There are four lists provided so you won't get confused:

• books you finished,
• books you are currently reading,
• books you want to read later,
• books you didn't finish.

You can use custom tags and filter through them.

A book can be added by:

• looking it up in the Open Library database,
• scanning its barcode,
• adding its details manually.

You can also view some cool statistics!

Links:

• Battletech: Mercenary's Star by William H. Keith
• Permaculture for the Rest of Us by Jenni Blackmore
• Piranesi by Susanna Clarke
• Battletech: The Price of Glory by William H. Keith
• The Urban Homestead by Kelly Coyne
• Battletech: The Sword and the Dagger by Ardath Mayhar
• Battletch: Warrior En Garde by Michal A. Stackpole
• Sisterhood of Dune by Brian Herbert and Kevin J. Anderson
• Fundamentals of the Wudang Sword Method by Scott M. Rodell
• Battletech: Warrior Riposte by Michal A. Stackpole
• The Cardio Code by Kenneth Jay
• Mentats of Dune by Brian Herbert and Kevin J. Anderson
• How to Grow More Vegtables by John Jeavons

Provide feedback on this episode.

What looked like a simple online scam quietly grew, over more than a decade, into one of the most sophisticated cybercrime operations the FBI had ever traced. Custom malware. Opsec off the charts. Fleets of infected computers mining cryptocurrency for someone else. Millions of dollars siphoned from victims who had no idea.

This is the story of Bayrob and the three men from Romanian who were behind it. And the long, strange road that led American investigators to their door.

Sponsors

Support for this show comes from ThreatLocker®. ThreatLocker® is a Zero Trust Endpoint Protection Platform that strengthens your infrastructure from the ground up. With ThreatLocker® Allowlisting and Ringfencing™, you gain a more secure approach to blocking exploits of known and unknown vulnerabilities. ThreatLocker® provides Zero Trust control at the kernel level that enables you to allow everything you need and block everything else, including ransomware! Learn more at www.threatlocker.com.

This show is sponsored by Meter, the company building networks from the ground up. Meter delivers a complete networking stack - wired, wireless, and cellular - in one solution that’s built for performance and scale. Alongside their partners, Meter designs the hardware, writes the firmware, builds the software, manages deployments, and runs support. Learn more at meter.com.

This show is sponsored by Maze. Maze uses AI agents to triage and remediate cloud vulnerabilities by figuring out what’s actually exploitable, not just what’s theoretically risky. They remove the noise, prioritize vulns that matter, and manage remediation, so your team stops wasting time on meaningless vulns. Visit MazeHQ.com/darknet for more information.

Support for this episode comes from NetSuite. NetSuite gives you visibility and control of your financials, planning, budgeting, and of course - inventory - so you can manage risk, get reliable forecasts, and improve margins. NetSuite helps you identify rising costs, automate your manual business processes, and see where to save money. KNOW your numbers. KNOW your business. And get to KNOW how NetSuite can be the source of truth for your entire company. Visit www.netsuite.com/darknet to learn more.

This episode is sponsored by Chainguard. Chainguard builds container images the right way — minimal, hardened, and built from source every single day. We’re talking images with zero known CVEs, designed from the ground up for production. No bloat. No mystery packages. No 2 a.m. patching marathons because some transitive dependency lit up your dashboard. Stop patching images that are insecure. Start shipping clean. Head to chainguard.dev to see how secure your software supply chain can really be.

From https://en.wikipedia.org/wiki/Simon_(game)

Simon is an electronic game of short-term memory skill invented by Ralph H. Baer and Howard J. Morrison, working for toy design firm Marvin Glass and Associates, with software programming by Lenny Cope. The device creates a series of tones and lights and requires a user to repeat the sequence. If the user succeeds, the series becomes progressively longer and more complex. Once the user fails or the time limit runs out, the game is over. The original version was manufactured and distributed by Milton Bradley and later by Hasbro after it took over Milton Bradley. Much of the assembly language code was written by Charles Kapps, who taught computer science at Temple University and also wrote one of the first books on the theory of computer programming. Simon was launched in 1978 at Studio 54 in New York City and was an immediate success, becoming a pop culture symbol of the 1970s and 1980s

Provide feedback on this episode.

New hosts

There were no new hosts this month.

Last Month's Shows

Comments this month

Past shows

• hpr4424 (2025-07-17) "How I use Newsboat for Podcasts and Reddit" by Archer72.
  
• أحمد المحمودي said: "How did I find HPR" (2026-05-12 17:16:07)
  
• candycanearter07 said: "Re: How did I find HPR" (2026-05-12 19:36:04)
  
• hpr4502 (2025-11-04) "Cheap Yellow Display Project Part 3: Reverse beacon network " by Trey.
  
• Archer72 said: "Morse code" (2026-05-19 15:00:39)
  
• hpr4567 (2026-02-03) "Movie Recommendations for Hackers" by Deltaray.
  
• Antoine said: "Some watched!" (2026-05-27 03:33:12)
  
• hpr4587 (2026-03-03) "UNIX Curio #1 - Shell Archives" by Vance.
  
• Dave Morriss said: "Great reminder! I had forgotten shar" (2026-05-07 18:18:05)
  
• Vance said: "Color printing" (2026-05-08 22:02:23)
  
• hpr4607 (2026-03-31) "UNIX Curio #3 - basename and dirname" by Vance.
  
• Vance said: "Correction" (2026-05-12 01:03:12)
  
• hpr4618 (2026-04-15) "Simple Podcasting - Episode 2 - Basic Filtering" by Whiskeyjack.
  
• Henrik Hemrin said: "Generating sine wave" (2026-05-14 19:59:46)
  
• Whiskeyjack said: "Reply to Henrik Hemrin on Sine Waves in HPR4618" (2026-05-15 15:22:16)
  
• hpr4627 (2026-04-28) "UNIX Curio #5 - Faster, Pussycat! Kill! Kill!" by Vance.
  
• Steve Barnes said: "Thanks for the context!" (2026-05-21 05:28:05)
  
• hpr4628 (2026-04-29) "Nuclear Power Technology Follow Up" by Whiskeyjack.
  
• Antoine said: "o/" (2026-05-17 00:24:23)
  
• Whiskeyjack said: "Reply to Antoine on HPR4628" (2026-05-17 18:08:18)
  

This month's shows

• hpr4631 (2026-05-04) "HPR Community News for April 2026" by HPR Volunteers.
  
• Whiskeyjack said: "Response to Ken Fallon in HPR4631 Community News" (2026-05-04 05:13:02)
  
• candycanearter07 said: "new episodes" (2026-05-04 18:47:39)
  
• Ken Fallon said: "new candycanearter07 episodes" (2026-05-05 09:00:00)
  
• أحمد المحمودي said: "Thanks for the encouragement" (2026-05-12 17:18:20)
  
• hpr4634 (2026-05-07) "Upgrade Failsause" by operat0r.
  
• YourName said: "Why is the audio so bad OMG sorry" (2026-05-06 20:05:43)
  
• Ken Fallon said: "Already reported and fixed" (2026-05-06 20:35:49)
  
• hpr4637 (2026-05-12) "UNIX Curio #6 - at and batch" by Vance.
  
• candycanearter07 said: "still useful!" (2026-05-13 03:43:37)
  
• norrist said: "at for scheduled reboots" (2026-05-16 19:41:04)
  
• Vance said: "Good points" (2026-05-17 03:03:47)
  
• Whiskeyjack said: "At and batch in HPR4637" (2026-05-17 18:35:11)
  
• hpr4640 (2026-05-15) "Robert A. Heinlein" by Ahuka.
  
• Antoine said: "Nice tips" (2026-05-17 00:22:01)
  
• Kevin O'Brien said: "I enjoyed doing it" (2026-05-17 19:14:19)
  
• hpr4646 (2026-05-25) "Mobile Gaming" by Elsbeth.
  
• candycanearter07 said: "my opinion of mobile gaming" (2026-05-27 00:24:58)
  
• hpr4647 (2026-05-26) "UNIX Curio #7 - Compression" by Vance.
  
• xmanmonk said: "Great Series" (2026-05-27 00:29:32)
  
• candycanearter07 said: "thoughts" (2026-05-27 13:20:13)
  

Mailing List discussions

Policy decisions surrounding HPR are taken by the community as a whole. This discussion takes place on the Mailing List which is open to all HPR listeners and contributors. The discussions are open and available on the HPR server under Mailman.

The threaded discussions this month can be found here:

Events Calendar

With the kind permission of LWN.net we are linking to The LWN.net Community Calendar.

Quoting the site:

This is the LWN.net community event calendar, where we track events of interest to people using and developing Linux and free software. Clicking on individual events will take you to the appropriate web page.

Provide feedback on this episode.

In our sample game we look at playing as Austria and aiming for a Diplomatic Victory. And our focus is on puppeting Citty-States, but be I misunderstood and instead of making a Diplomatic Victory easier, it makes it harder. I still managed to get my Diplomatic Victory, but a Science or Domination Victory would definitely have been easier in this scenario.

Playing Civilization V, Part 12

A Diplomatic Victory Strategy

Civilization V introduced a new Victory type and I thought it might be fun to try this strategy for a sample game to see broadly how this would work. I decided I would play as Austria on Prince level, which means that all players are equal and no one is favored. For my map I chose Fractal, Map Size = Standard, Game speed = Standard. My only Advanced Option was Quick Combat, because I didn’t need to see that drawn out, particularly since I plan to avoid combat as much as necessary. My leader is Maria Theresa, and her Austria has a unique ability called Diplomatic Marriage. This allows us to either Annex or Puppet any city-state that has been allied to us for 5 turns with the proper expenditure of Gold. My plan was to use this to Puppet the city-states to control their votes for the Diplomatic Victory, but that was a misunderstanding. I actually implemented one of the hardest ways to win a Diplomatic Victory. But that is what I did. Now to get them to ally with me the most effective way to do this is with cash and lots of it, and of course even more cash to actually effect to Puppeting of them. So my overriding objective in this game is to amass a large Treasury. But of course I cannot ignore my military either, since a weak military invites attacks form greedy neighbors. And I may need to “liberate” the occasional city-state if another player conquers them. Austria also has a Unique Unit, the Hussar, which replaces the Cavalry unit. It can move after attacking, has a flanking bonus, and has one extra movement. And the Unique Building is the Coffee House, which increases the generation of Great People in the city by 25%. So you can expect me to build these in every city as well.

With that in mind, I started the game and settled in place, I had Mountains nearby, but also Sheep and Silver within my city, so some useful resources. I immediately started to produce a Scout as my first unit, and sent my Warrior out to explore. In the very early stage I focus on exploring the surrounding area and finding any Goody Huts, i.e. Ruins. My initial city site is not exactly ideal, as it is all hilly with Jungle nearby. After building my two Scouts, my next priority was to build a Worker unit to increase the productivity of my city. And for my first social policy I unlocked Tradition. When I got my second policy I picked Oligarchy, and plan to complete all of the Tradition tree. I cleared out a Barbarian encampment, and then discovered my first City-state, Vilnius. Then I needed to clear out another Barbarian encampment that was blocking me from finishing my exploration. Meanwhile I am focusing on getting techs for sailing the ocean blue, because that is how I plan to get trade routes, discover City-states I can puppet, and so on.

Because money is key to my strategy I made a beeline for Currency in the my Science research. And while headed there I completed the Tradition tree. When I can get there I will work on the Commerce tree to maximize my cash, but until then the Patronage tree will let me improve my City-state relations, which is important for improving my relationships with City-states. After all, you need to be allies with them for 5 turns before you can puppet them. I now have three cities, and can probably squeeze out a few more, which should be sufficient to my needs.

At Turn 141 I have 6 cities, which is all I will build in this game. I now have the technology to build Workshops, which are the first productivity boosters available, so I set all my cities to building them. For Research my immediate object was to get to Compass so I could build the Galleass, which would let me do more ocean exploration. But to go into deep Ocean I will need to go further to get Astronomy, which will let me build the Caravel which can enter deep ocean. Once I got that I switched to Banking, not just for the money, but as a prerequisite to building the Forbidden Palace, which grants two additional delegates in the World Congress/United Nations. When I get Banking I’ll go back to Astronomy, and then Navigation, to advance my seagoing capabilities.

By Turn 216 I had gotten Banking and started on the Forbidden Palace. And by luck, just as I got started I got a Great Engineer. I am holding him in reserve in case I need to hurry production, since Great Engineers are the only way to do that in Civ 5. I also picked up Astronomy, which will let me build Caravels to explore the whole ocean. My next research priority will be Gunpowder since it is time to beef up my defenses.

At Turn 240 my Caravels started to come out, and I found several new City-States. And since my Treasury is healthy (I started with 6,000 gold, and I’m bringing in 100 each turn), I began the process of puppeting the City-states.

Puppeting City-states as Austria

Let’s look at this in detail since it is important. The requirements are 2 things:

• Be allied with the City-state for 5 consecutive turns
• Have the cash needed. This amount is not too much early on, but it rises over time.

So, how do you become allies with a City-state? There are a number of things you can do to improve your relationship. You can take on a quest that a City-state has published, which can be things like “Find another Natural Wonder” or “Create a Great Admiral”. These quests pop up continuously throughout the game, and you are free to ignore them, but fulfilling one will improve your relationship. Trade will also improve your relationship, so in this game all of my Trade routes were made with City-states. If you are in a position to have a successful war, you can find a former City-state that was conquered by one of the other Empires, liberate it, and then it will be your ally for the rest of the game. Giving them presents is how I usually do it, though. You can give them units or money. If your purpose is to get allies, money works best. But I do gift units in 2 circumstances. First, if I have obsolete units, giving them away might be better then deleting them. Second, if a City-state is under attack by another Empire. Gifting them units might help them hold out and make life difficult for a rival.

The place where all of this is done is the City-state screen which opens up when you click on the bar above the City-state, which is where you handle all of your relationships. At the top of this screen you see your current status, which more often than not will be Neutral, which is how all City-state relationships start out. But you can get them angry by, for instance, moving a unit of your into their territory. If you only do it once, and give them time to get over it, they will go back to Neutral. You can also improve the relationship by pledging to protect them, but be careful. If they get attacked and you do not try to protect them, they will get very angry. Giving a gift opens a pop-up to say what kind of gift: 250 gold, 500 gold, 1000 gold, or a Unit. Note that a Unit is only worth 5 influence points, while 250 Gold is worth 20, so as I said Money is more powerful if your aim is to improve your status with them.

So at Turn 242 I found the City-state of Singapore, and it appears that I was the first Empire to find them. So I immediately pledged to protect them. My Influence with them was 20, which is Neutral. We just met, and that is where things stand on first meeting. The be Friends you need to get to 30, and to be allies you need to get to 60. When I clicked Next Turn, I got a Quest from Singapore. They were worried about a Barbarian Encampment nearby, and if I cleared it out I would get additional influence with them. In this case, though, I let that go by. I want to move more quickly, and Singapore is across the Ocean from me. My influence with them at this point was 21, so only a modest increase. But I have 6038 Gold in my Treasury, and I am bringing in 102 per turn. So let’s see what a gift of 500 Gold will do. It brings me to 65 influence, so we are now Allies. But when I mouse-over Singapore, the pop-up window reminds me that my Influence will decrease by 1.12 per turn. For an Empire other than Austria this would mean a regular infusion of cash to keep up your status. And I have won Diplomatic victories with other Empires by saving up a lot of cash and dumping it on City-states just before the United Nations vote.

But for Austria you have special ability called Diplomatic Marriage that lets you turn the City-state into a Puppet, and that is permanent. But it also presents some obstacles as we will see. We are allies now, but my influence will drop by 1 each turn, and I might lose my allyship before I can puppet them. But I can gift a unit and get another 5, and I happen to have a very obsolete Warrior unit that will serve the purpose. However, it takes three turns for the Unit to arrive, so I lose few more points. At Turn 247 I can now use the diplomatic Marriage option to make a puppet of Singapore. Prior to doing this I was fourth in score with 604, while the leader had 729. My Happiness Level was +25, and I now had 5833 Gold in my Treasury. Then I made Singapore a puppet, and now I am third in the game with a score of 664. My Treasury has fallen to 5258, which means it cost me 575 Gold. But most significant is that my Happiness fell from +25 to +9, which is a huge loss. I have enough gold to puppet 4-5 more City-states at this time but if I did I would have rebellions breaking out and my Empire would eventually collapse.

This is the obstacle that Austria has to face. We need to promote Happiness before we go much further with making puppets. Civ is always a game of balances.

Links

• https://civilization.fandom.com/wiki/City-state_(Civ5)
• https://civilization.fandom.com/wiki/Austrian_(Civ5)
• https://www.palain.com/gaming/civilization-v/playing-civilization-v-part-12/

Provide feedback on this episode.

It is suggested reviewing the episode What did I do at work today? Part 3 Section 1 prior to listening

Test driven development - a way of writing code that involves writing an automated unit-level test case that fails, then writing just enough code to make the test pass, then refactoring both the test code and the production code, then repeating with another new test case. - https://en.wikipedia.org/wiki/Test-driven_development

Joplin - Joplin is an open source, cross platform note-taking app. - https://joplinapp.org/

PHP - A popular general-purpose scripting language that is especially suited to web development. Fast, flexible and pragmatic, PHP powers everything from your blog to the most popular websites in the world. - https://www.php.net/

MySQL - MySQL is an open-source relational database management system. MariaDB is a community developed fork of MySQL, often installing the MySQL package on a Linux distribution will actually install MariaDB. - https://en.wikipedia.org/wiki/MySQL - https://mariadb.org/ - https://www.mysql.com/

Sublime Text - Cross platform text editor - https://www.sublimetext.com/

Nmap - Network Mapper is a free and open source utility for network discovery and security auditing - https://nmap.org/

Markdown Fenced code blocks - "A code fence is a sequence of at least three consecutive backtick characters (`) or tildes (~). (Tildes and backticks cannot be mixed.) A fenced code block begins with a code fence, preceded by up to three spaces of indentation. The line with the opening code fence may optionally contain some text following the code fence; this is trimmed of leading and trailing spaces or tabs and called the info string. ... Although this spec doesn’t mandate any particular treatment of the info string, the first word is typically used to specify the language of the code block."

```ruby
def foo(x)
 return 3
end
```

from CommonMark Spec at https://commonmark.org/ (CommonMark is a standard, interoperable and testable version of Markdown.)

Writing to a Database with PHP

The following PHP method is implemented within a database access class:

function create_with_id($id, $name) {
  $born = time();
  $id = mysqli_real_escape_string($this->db, $id);
  $name = mysqli_real_escape_string($this->db, $name);
  $sql = "INSERT INTO object
      (object_id, display_name_text, born, died)
    VALUES
      ($id, '$name', $born, 0);
  ";
  db_run_sql($this->db, $sql);
}

Note db_run_sql is a helper function defined elsewhere, not a built in function, and the property db is a previously initialized mysqli object.

Provide feedback on this episode.

01

This is the fourth episode in a four part series on simple podcasting.

02 Introduction

In this episode we will discuss alternatives to Audacity when it comes to analyzing audio spectrums to find the sources of unwanted noise.

I previously promised some gratuitous hackery, and we will get into that in this episode.

03

Recall that with Audacity you first import the audio file, then select the part of the audio you wish to analyze (or ctrl-A for all), and then select analyze > plot spectrum.

This is in fact the only feature of Audacity that I know how to use. I am definitely not an audio expert.

I do however have some background in processing and analyzing other signals, so some of the basics are familiar to me.

04

We can accomplish the same thing that Audacity does in this instance provided we can do the following.

First, we need to get the data out of the audio file and into a form which we can import into other software.

Second, we need to perform certain mathematical operations on this data.

Finally, we need to be able to plot the results of these calculations on a chart.

--------------------

05 Fourier Transforms

First though, we need a bit of mathematical background.

What Audacity is doing when it shows a plot of frequency versus amplitude is that it is showing the results of a Fourier Transform.

A Fourier Transforms is a mathematical operation that converts the time domain into the frequency domain.

Any complex signal, audio or otherwise, can be broken down into a collection of sine waves of various frequencies.

For example, a simple square wave signal of say 100 hertz can be represented as a sine wave of frequency 100 hertz plus a collection of higher frequency sine waves which add together to give the sharp corners.

06

A Fourier Transform finds these sine waves and sorts them out into separate bins, with each bin representing an individual frequency or a collection of closely related frequencies, depending on how fine grained the sorting is.

07

This is exactly what we want when we are trying to figure out how to filter out noise.

Recall that earlier in this series we had to solve a problem with a high pitched background noise which was originating in my cheap microphone.

Analyzing this audio by frequency showed that it was a series of individual tones at 1 kHz intervals.

We were then able to use filters targeted at those frequencies to get rid of that noise.

08

There are several optimized versions of the Fourier Transform algorithm.

A very common one is the Fast Fourier Transform, common abbreviated to just "FFT".

This is so common that the term "FFT" is often used to simply mean any Fourier Transform even though this is not technically correct.

09

Typical FFT algorithms require that the number of data samples is exactly a power of two.

So the number of samples we need may be something like 4096, 8192, or 65536, to give a few random examples.

When we transform from the time domain to the frequency domain, each sample becomes a single frequency "bin". So the more samples we have, the finer the resolution we get in terms of frequency.

10

If we assume we are dealing with flac files recorded at a 44.1 kHz sample rate, that is, 44100 samples per second, then if we have 32768 samples, each "bin" represents slightly more than 1 hertz.

If we have 65536 samples, then each "bin" represents a fraction of a hertz.

For our purposes we will pick 65536 samples.

That means we need 1.48 seconds of data.

For simplicity's sake we will record at least 2 seconds of data and then just discard the samples that we don't need.

11

There is a further complication here. Fourier Transforms normally work with complex numbers.

Recall from your school days that as well as integers and real numbers there are complex numbers.

Each complex number consists of two parts, a real component and an imaginary component.

I won't go into the details of this, just accept that each sample needs to have two components.

Fortunately, if we don't have complex number data we can just set the imaginary component to zero and use that.

This is enough talking about the theory, let's get into the practical details.

--------------------

12 Extracting Data from Audio Files

First we will look at how to extract the data from the audio files.

Fortunately, one of the programs which we have already been using can do this.

To do this we will use Sox.

I am not aware of an equivalent feature in ffmpeg.

13

Sox calls itself "SoX - Sound eXchange, the Swiss Army knife of audio manipulation"

Sox is free software and is licensed under the GPLV2 or later.

In this case we want to use a feature which allows us to convert a binary audio signal file to a text data file.

To convert the file to text data we just give the output file a ".dat" file extension and Sox will do this for us.

14

Here is a command example.

sox inputfile.flac tdata.dat

15

This gives us a file in the following format, assuming this is a mono audio recording.

; Sample Rate 44100

; Channels 1

0 0.045471191406

2.2675737e-05 0.055023193359

4.5351474e-05 0.048217773438

6.8027211e-05 0.053192138672

etc.

The first line states the sample frequency

The second line states that the data is for channel 1.

The data starts on the third line.

Column 1 is the time in seconds.

Column 2 is the waveform data point.

16

To analyze the data we want a subset of these samples.

When we convert from the time domain to the frequency domain, our resolution will be determined by the number of samples. We would like therefore to have at least as many samples as the sampling rate.

We also want the samples size to be an even multiple of two.

The number of points we want to have is equal to the next even multiple of two above our chosen sampling rate, 44,100 Hz.

This number would be 65536.

17

To extract this data from the file we can do the following.

tail tdata.dat -n+3 | head -n65536 | awk '{printf "%s\n", $2}' > tdata.csv

18

We use tail to skip over the first three lines.

We use head to take the next 65536 lines and discard the rest.

We use awk to extract the second column which we will use as the real component.

We now have this data as a csv file in one column.

--------------------

19 Analyzing the Data

To analyze the data we need software which can calculate FFTs.

I will now show two examples of this, a very simple case using Libre Office Calc, and a more complex but more complete one using GNU Octave.

20 Using Libre Office

We can do fourier analysis and plot charts using Libre Office.

Take the csv file of data that we previously created.

For this example I used data from a recording of silence so that I could see what internal noise was being generated by the headset.

Open the csv file and import it into Libre Office Calc.

21

Now select all 65536 rows of column A.

The Fourier function will automatically fill the imaginary component with zeros if we don't provide an column of imaginary numbers, so we don't need to provide a column of zeros.

Then select Data > Statistics > Fourier Analysis.

22

A window will open allowing you to select various parameters.

For Results to:, enter "D1".

Grouped by Columns.

Select OK.

23

New data should now appear starting in cell D1.

The first line will say " Fourier Transform"

The second line will state the input range.

The third line will state "Real" in column D, and "Imaginary" in column E.

The data will start in row 4.

24

For our simple example we will ignore the imaginary data and just use the real data, which will form our Y component when we plot it on a chart.

We now need to create the X axis data.

25

Each cell is a "bin" of frequencies.

Each cell therefore represents (sample frequency) / (Number of samples) Hz.

26

To create the X axis data showing frequency, enter the following formula in to column C to the left of each D column number.

=((44100/65536) * (ROW() - 4)

27

We can now create an XY chart showing the frequency analysis.

You may need to exclude the first couple of dozen rows as very low frequency components which cannot be heard may otherwise overwhelm the data we are interested in.

Also, you only need the first half of the chart.

The FFT mirrors the data from the first half of the array into the second half.

28

Because characterizing a sine wave requires a minimum of 2 points, although we have a sample frequency of 44.1 kHz, we really only have sound waves up to a maximum of half that, or 22.05 kHz.

Create the chart with lines only.

If you followed the above instructions, you should see something resembling what we saw in Audacity, except with each bin more sharply defined.

29

In the data that I had from a recording of unfiltered headset noise, I could see a distinct noise spike every 1000 hertz.

30

However, we have taken several shortcuts.

First, the imaginary component of the data was ignored.

Second, the magnitude (that is, Y axis) has both positive and negative peaks.

Third, the data is not scaled to dB sound units, so we just have a relative measure.

However, that by itself is enough to tell us where the frequencies are that we need to construct filters to deal with.

31

We could refine this spreadsheet a bit more to deal with the above issues, but I think we have demonstrated the basic principle, and working with a spreadsheet can be a bit awkward.

However, if working with a spreadsheet is what you want to do, then you can add more columns and more formulae to improve on it.

--------------------

32 Other Analysis Software

I will go on to GNU Octave in a moment, but I want to get a few other alternatives out of the way first.

I won't go into any detail on them other than to point them out to people who want to have a go at trying these themselves.

33 Grace

There is math and plotting software called Grace.

This is free software, released under the GPL V2.

According to the documentation, it seems to have the features we need, including an FFT function.

However, I could not get it to work properly on Ubuntu 24.04.

I could not get it to load a data file and plot data.

34

The error messages were vague and unhelpful.

The file navigation system didn't work.

There was no obvious path to success, and if it isn't easy to use then there is no point to it.

This is fairly old software, designed for X Window and Motif.

I gave up on it as not suitable for this series as I am looking for some fairly low effort things for people to try themselves.

If someone else can get it to work on their PC, perhaps they could do an HPR episode on this themselves.

35 Command Line FFT Packages

There are several command line FFT packages.

They will read data from std in or from a file and output the FFT.

However, these are not packaged for Ubuntu and appear to be distributed as C source code which you would download and compile.

You can experiment with those if you wish, but I felt they were a bit out of scope for discussion here as I am looking at common tools that are ready to use.

36

Here are two examples.

One is

Command-line Fast Fourier Transform utility

https://github.com/gregfjohnson/fft

Another is

cli-fft

https://github.com/jonolafur/cli-fft

37

I have not tried these and cannot say whether they are any good or not.

Similarly, there are a number of FFT packages that are libraries for languages such as Python.

If you want to take the time to write a short program to go with them, you can create a dedicated FFT command line program.

However, I felt that this too was out of scope for what I was trying to do here.

38 Doing it the Hard Way

Hypothetically, it may be possible to write an FFT function in bash bc, which is the arbitrary precision calculator language which is part of the standard shell package.

I say hypothetically, because I have not tried it.

I think it would be an interesting challenge, but I don't have the time at the moment to try it.

If anyone feels motivated to give it a try, they're welcome to give it a go and then do a podcast episode on it.

--------------------

39 GNU Octave

We have seen that as well as using features built into Audacity to analyze the audio spectrum to see the frequencies of undesired noises, we were able to do the same using a Libre Office spreadsheet.

40

Now we'll look at another bit of software, GNU Octave. GNU Octave is free software, licensed under the GPL V3 or later. It is a mathematical scripting language, very similar to Matlab. People use it for mathematical, engineering, and scientific work. It can be found in most Linux distros and is available for some other operating systems as well.

41

Octave has two features built in that we need for our purposes. It does FFTs, and it has a plotting system built in to produce graphs.

--------------------

42

We will take the same audio test file that we used with Audacity and Libre Office and use it here as well.

The bash script to convert the flac file to text data is essentially the same, with the exception that file extension on the output file as is ".txt" instead of ".csv". This latter change was an arbitrary decision on my part.

43

As a quick review, this bash script uses sox to convert a flac file to a text ".dat" file.

Then it uses tail, head, and awk to extract the first 65536 rows of data, skipping over the header information and ignoring the first column of time data.

This script will be in the show notes.

--------------------

#!/bin/bash

# This version is for use with the GNU Octave script.

sox hsnoisemono.flac hsnoisemono.dat

tail hsnoisemono.dat -n+3 | head -n65536 | awk '{printf "%s\n", $2}' > hsnoisemono.txt

--------------------

44

We now have a 1.1 MB file containing 65536 samples of data in text format.

Now the next thing we need to do is to create a short Octave script file.

I will just give a brief overview of the script here, the full script will be in the show notes.

45

I put the script in a file called "octavespectrum.m". I have never used Octave before now, but the convention seems to be to give the script a ".m" ending.

The "she-bang" line is "#!/usr/bin/env octave". If you make the file executable you can run it like any other script, or you can type "octave" and then the name of the script to run.

46

I won't read out the script in detail, as that would be too hard to following along in a podcast.

However, I pass several arguments to the script including the name of the data file, and then two integers that I use to limit the display area in the Y and X axes so I can have the chart focus on the areas of interest that I want to see.

I also pass a string containing the name of the graphic file that I want the chart exported to.

This was an arbitrary decision on my part and you can just hard code these values in if that is what you want to do.

47

The arguments are accessed by calling the "args()" function, which returns an array of strings.

Next, it reads in the specified file using the "dlmread()" function. This reads all of the data into an array.

48

Next, it performs a hamming windowing function on the data. I'll explain that briefly.

It is standard practice when doing FFT signal processing to "window" the signal.

Since the signal sample is of finite length, it will stop at each end of the array.

49

Unless you were lucky enough for this to happen exactly at a zero crossing, this would produced an abrupt transition in the data which looks like "noise" to the FFT.

The solution is to taper the signal off gradually towards the ends so that when it gets cut off the signal is fairly small at that point anyway.

There are a variety of different windowing functions, but "hamming" seems to be the most commonly used.

50

Next, it does an FFT using the "fft()" function.

51

This gives us real and imaginary outputs.

These are combined by summing the squares of each corresponding real and imaginary element and then taking the square root of each and storing that in a new array. This gives a single array of the same length as the originals, but combining the two output components.

If anyone wants to tell me that this isn't how things are done in the audio world, they're welcome to make an HPR episode telling us all the right way to do things.

52

Then it does some scaling and selection of subsets of data so we get the X axis in hertz and just the number of samples that we wish to look at.

If you are looking at the script, the thing to keep in mind is that Octave will work on entire arrays of data in a single operation. You don't need to write explicit loops for this. The looping is handled implicitly as part of the syntax.

53

It also does various other things that make the chart easier to read. The comments in the script describe these in more detail. Since this is a script it's easier to add these sorts of refinements than is the case for a spreadsheet so I have made the effort to add them.

Finally it calls the "plot()" function.

If an output graphics file name was provided, it also creates a PNG file containing the same image using the "saveas" function.

54

We now see the chart, and it looks more or less as expected.

However, this chart is interactive.

You can zoom and pan the data, something that you can't do with either Audacity or Libre Office.

The chart window doesn't have a function for exporting the resulting chart to a "png" file, it will only save to an ".ofig" file. The ofig file is not a standard graphics file, it is a serialization of the chart data that can only be looked at using the Octave chart viewer.

55

Alternatively, you can just take a screenshot of the chart after you have interactively zoomed and panned to a point of interest.

At the bottom left of the chart window is a pair of x-y coordinates which tell you the current position of the mouse pointer in chart units.

This is very handy as it can be used to get the exact (or close to exact) frequency of each noise spike.

56

The Y axis is not scaled in any particular units such as dB, as I'm not sure how to do that according to audio industry conventions.

On the other hand, I'm not sure that it's really necessary, as I don't know what dB means in tangible terms anyway.

It does show relative sizes, so it helps to determine whether you have one noise frequency or multiple frequencies to worry about.

57

If anyone is familiar with how to scale the raw data from a flac file as exported by Sox into dB units according to audio industry convention, then they are welcome to create an HPR episode telling us how to do it.

--------------------

58 Comments on GNU Octave

I had never used GNU Octave before this, although I had heard of it and it is quite a significant piece of software for a specific segment of users.

59

The syntax is a bit odd especially in how it deals with array operations, but I was able to google various examples and answers to eventually get this working.

A few other peculiarities are that it uses the percent "%" character to denote a comment, and leaving out the semi-colon at the end of the line causes it to print the answer to the console after executing the statement.

60

The GNU Octave solution was harder to get working than the Libre Office method.

However, once it was working it is easier to use repeatedly.

If I were to want to automatically generate audio files with different filtering or other options and wanted to script the creation of a large number of images showing the results, this would be the way to do it.

61

When your run the Octave script you may get a warning which says something like

"QSocketNotifier: Can only be used with threads started with QThread".

This is apparently a routine warning message from the Qt graphics system which has no real significance in this context and can be ignored for our purposes.

--------------------

62

We now have a bash script which will use sox to extract the data from a flac file, and a GNU Octave script which can be used to display the resulting frequency spectrum.

This does more or less the same thing as "Plot Spectrum" does in Audacity, but allows for zooming and panning to get a more detailed look at the data.

63

However it doesn't give you an absolute reading of the sound levels in dB, something that Audacity does provide.

What I wanted it for though was to find the frequencies of the audible noise in the signal, something that it does quite well.

--------------------

#!/usr/bin/env octave

% Perform an FFT on the data in a file and plot the results.

% ======================================================================

% The sampling frequency. This must be changed to accommodate the

% actual sampling frequency if it was something else.

samplefreq = 44100;

% Thickness of line on plot.

linewidth = 2;

% ======================================================================

% The name of the data file is passed as a argument.

args = argv();

if length(args) < 3

quit

endif

% File name.

fname = args{1};

% Clip the peak values.

peakclip = str2double(args{2});

% How much data to show, in kHz.

rbound = str2double(args{3}) * 1000;

% The optional file name to save a chart image to.

if length(args) > 3

chartfile = args{4};

else

chartfile = "";

endif

% ======================================================================

% Read the data in from the file.

sampledata = dlmread(fname);

% Number of samples.

samplecount = length(sampledata);

% ======================================================================

% Window the data. This helps deal with the discontinuity of data at

% each end of the array and the effects this has on introducing apparent

% noise into the signal.

windoweddata = (hamming(samplecount) .* sampledata);

% ======================================================================

% Do the actual FFT.

fftresults = fft(windoweddata);

% Get real component.

r = real(fftresults);

% Get the imaginary component.

i = imag(fftresults);

% Combine the real and imaginary. In order to square each element of each

% array, we must use the ".^" operator, not just "^".

rfft = sqrt(r.^2 + i.^2);

realfft = rfft(1:samplecount);

% ======================================================================

% Scale factor for frequency.

fscale = samplefreq / samplecount;

% X axis scale, scaled to frequency.

f = (0:samplefreq/2) * fscale;

% Take a subset of the data if specified. rbound has to be re-scaled

% from kHz to array increments.

freq = f(1:min(rbound / fscale,length(f)));

% y axis. We take the absolute value and then limit (clip) the peaks

% so that a few large peaks don't obscure the smaller ones.

mag = min(abs(realfft(1: length(freq))), peakclip);

% Plot the results.

figure;

whandle = plot(freq, mag, 'LineWidth', linewidth);

title(["Audio Spectrum of ", fname]);

xlabel("Frequency (Hz)");

ylabel("Unscaled Magnitude");

grid on;

% If the appropriate optional argument was specified, save the chart

% to a file of that name.

if length(chartfile) > 4

saveas(gcf, chartfile, "png");

endif

% Need this so the plot window stays open.

waitfor(whandle);

% ======================================================================

--------------------

This is the shell script used with the above Octave script.

The arguments are

1 - the file name for the input data file.

2 - The value to clip the peaks at.

3 - The upper frequency bound in kHz.

4 - The output graphics file name.

#!/bin/bash

octave octavespectrum.m hsnoisemono.txt 10 12 hsnoisemono.png

--------------------

64 Episode Conclusion

In this episode we covered the following topics.

What Fourier transforms are.

Extracting data from audio files using Sox.

Analyzing the data using Libre Office.

Analyzing the data using GNU Octave.

And, several alternative analysis methods.

65 Series Conclusion

This is the end of a four part series on simple podcasting.

In the first episode, we covered a simple podcast recording method. This first episode is all you really need to make a podcast.

66

In the second episode we covered basic filtering and a few other simple topics. The methods discussed in that episode provide basic improvements to your audio if you feel the need for it.

67

In the third episode we covered how to analyze audio noise problems using Audacity and additional filtering techniques to deal with specific problems that we may find.

We also covered command line recording, playback, and getting information about an audio recording.

68

In the fourth episode we engaged in a bit of gratuitous hackery for the fun of it and showed how to use alternative software methods to analyze audio signals.

69

I hope that this series has been both useful and entertaining and that you will use the knowledge gained here to create and submit your own HPR podcast episodes.

--------------------

--------------------

Provide feedback on this episode.

This series is dedicated to exploring little-known—and occasionally useful—trinkets lurking in the dusty corners of UNIX-like operating systems.

In UNIX Curio #4 ( HPR episode 4617 ), I teased the subject of file compression. Today I'm circling back to that.

The history of data compression goes back at least to the 1970s, and in contexts outside UNIX and computers, probably even earlier. Somehow, it is refreshing to learn that humans have always struggled to have enough storage space to keep all the data they want to hang on to. One way around this limitation is to use some form of compression.

I am only going to dive into lossless compression for this episode—that is, a compression method that can be reversed and will spit out the original data bit for bit. Lossy compression methods also have their places: you might be familiar with their use for audio (such as Ogg Vorbis or MP3); it's also used for images (such as JPEG). Lossy compression allows some of the original data to be thrown away, resulting in a smaller file than is possible with lossless compression, but the intent is for the result to still sound or look "good enough" to a human observer. Also, I am going to limit my discussion to generic methods used for many types of data; while FLAC does lossless compression, it is specifically designed just for audio.

I should make clear that I have never studied computer science or information theory, so this episode will not get into the science behind various types of compression algorithms and how they differ. But in general, these methods take advantage of the fact that many types of data have recurring patterns. English text mostly consists of words that often re-appear many times—source code similarly has keywords and variable names that recur. Compression is accomplished by representing a piece of data that occurs multiple times with a symbol that is shorter in length.

The first compression program in the UNIX world I could find is called pack , from 1978 ¹ . It was shortly followed in 1979 by a similar program called compact ² . Both of these used a technique called Huffman coding, but with some differences between them. Files compressed with pack were given a .z extension and compact gave filenames a .C extension. Roughly every five or ten years after this, a new program would come along and achieve lasting popularity.

There were, and still are, two opposing forces facing any new form of compression. Working in favor was the advantages it provided—first among these was achieving a better compression ratio, but performance improvements such as speed or reduced memory usage could also be compelling. The force against any new method was the fact that it was not yet widely supported—it doesn't much help to have a smaller file if the people you share it with cannot decompress it.

The next major advance in compression arose out of three scientific papers: two in 1977 and 1978 by Abraham Lempel and Jacob Ziv (called LZ77 and LZ78), and one by Terry Welch in 1984 which built on LZ78. This last method is typically referred to as LZW. Our UNIX Curio for today is a program called compress ³ that implements the LZW method. Files compressed this way are named with the extension .Z . I had always assumed that this was to honor Jacob Ziv, but now that I've researched the history, it seems more likely to be a follow-on from how files compressed by pack were named. Since pack did not use any of the Lempel-Ziv methods, I would guess that it used .z because that wasn't already taken by anything else, but that's pure speculation.

I do recall encountering .Z files in the wild, but feel certain that hasn't happened in the last 25 years, maybe longer. If you need to expand one of these, uncompress ⁴ is the program to use ( GNU's gunzip can also handle them ⁵ ). However, there was a serious problem that arose with the LZ78 and LZW compression methods. Both of them were patented, and the owner became aggressive in seeking payment from developers and users. The compress utility was developed within two months of the publication of Welch's 1984 paper and was included in Bell Laboratories' Eighth Edition UNIX before these shakedowns started. The paper did not disclose that a patent had been filed, and apparently Spencer Thomas and the other developers of compress were unaware of it. The utility became popular for a while, and was even standardized by POSIX, but people moved away from LZW once the legal threats started.

Another important advance came in 1991 and was called the DEFLATE compression method. It combined the un-patented LZ77 method with Huffman coding to achieve a similar level of compression as LZW (actually, often better) without the legal trouble. DEFLATE was developed for PKZIP and was soon adopted by the GNU project's gzip compressor. While Phil Katz (the "PK" in PKZIP ) patented one way of implementing the DEFLATE method, it was possible to write a compressor and decompressor without infringing ⁶ ; also, he apparently never tried to enforce the patent ⁷ .

As I mentioned in UNIX Curio #4, .zip is both an archive and a compression format. Each archive member can be compressed with one of several possible methods (or stored without compression). Unlike a tar file where compression can be applied to the entire archive, in .zip each archive member is compressed individually. This often means a .zip file will be slightly bigger than a tar file with the same contents compressed with gzip , because the .zip format cannot take advantage of duplication that occurs among more than one member of the archive. The vast majority of .zip files use only the DEFLATE and uncompressed storage methods and these are the only options if you want to follow the profile standardized in ISO/IEC 21320-1. Actually, since they both use DEFLATE, gzip is able to extract a .zip file in the special case where it only holds one member compressed with that method.

From the 1990s onward, people paid significant attention to avoiding patent landmines, so only methods that didn't have that problem became broadly popular. While the patents on LZ78 and LZW have since expired, I feel like their most successful legacy was in discouraging people from using those methods, leading to DEFLATE taking the popularity crown.

The next step came in 1996 and 1997 with the development of bzip and bzip2 by Julian Seward. The original method was quickly followed by bzip2 , which was the version that achieved true popularity. They use the Burrows-Wheeler transform, which does not itself compress data but re-arranges it to make it more compressible; this is combined with other techniques ⁸ . (At least, that's my understanding. I told you, I'm not up on information theory.) This provides a significant reduction in the compressed size of the data compared to earlier methods—however, it is slower than DEFLATE both during compression and decompression.

Separate projects have developed parallel versions of gzip and bzip2 that can take advantage of multi-processor machines, but the original utilities run single-threaded.

Another five years later, in 2001, Igor Pavlov added the Lempel-Ziv-Markov chain algorithm (LZMA), an enhancement to LZ77, to his 7-Zip compression tool. This was followed a few years later by LZMA2, a container format that allowed for LZMA compression to be split between multiple threads. Broad LZMA2 support came to the UNIX world in 2009 with the xz utility ⁹ . It offers roughly similar compression ratios to bzip2 , though it can be better or worse depending on the data to be compressed. While compression generally takes even longer than bzip2 , decompression is significantly faster (though still not as fast as gzip ). The Linux kernel relatively quickly supported booting from xz-compressed images ¹⁰ because it was a good match for that use case—compression, the time-consuming activity, only has to be done once while the more frequent decompression during boot happens relatively fast.

The last method I will cover is Zstandard ¹¹ , often written as zstd . This came about in 2015, and is another variation on LZ77 that uses finite-state entropy (which means nothing to me, but you might understand it). It performs about as well as DEFLATE in terms of compression ratios, but is much faster both when compressing and decompressing data. I should say that these statements are true with the typical default settings—depending on the compression level selected, it can compress more slowly, but compress the data smaller. However, decompression is always speedier than DEFLATE. This makes it attractive for some uses, and it is heavily promoted by Meta/Facebook, where Yann Collet developed it. For example, shipping large amounts of actively-used data between machines in a data center can go more quickly when the size is reduced; however, if the compression and decompression steps take too long that benefit is lost. A speedy method can be valuable even if it doesn't result in the greatest reduction in size. This use case stands in contrast to, say, a compressed backup file which might only be accessed in a disaster recovery scenario or never accessed at all, making size more important than speed.

Both the xz and zstd utilities have some built-in support for multi-threading, but the default is to run in a single thread. While xz can use multiple threads for decompression (but only if the file was compressed in multi-thread mode), the reference zstd utility can only use more than one thread for compression, not decompression.

There are many other methods of lossless compression that have been developed over the decades, but I believe these are the ones you are most likely to encounter in the world of UNIX-like systems. This is a personal opinion, and others might choose a different set. As mentioned, it can be tough for a new method to gain popularity and 35-year-old DEFLATE is still probably the most commonly used despite not being the fastest or offering the greatest reduction in size. Even systems like FreeBSD, NetBSD, and OpenBSD that do not like to include GNU tools supported it by developing their own version of gzip based on the permissively-licensed zlib library.

Technically, the LZW method used by the compress utility is still standardized by POSIX, so one might expect it to have the widest support. However, aggressive patent enforcement discouraged adoption, especially by Free and Open Source Software systems—even though the patent has expired, it is still out of favor compared to DEFLATE. For this reason, I feel justified in calling it a curio.

References:

1. Eighth Edition UNIX pack.c https://www.tuhs.org/cgi-bin/utree.pl?file=V8/usr/src/cmd/pack/pack.c
2. 2.9BSD compact.c https://www.tuhs.org/cgi-bin/utree.pl?file=2.9BSD/usr/src/ucb/compact/compact.c
3. Compress specification https://pubs.opengroup.org/onlinepubs/009695399/utilities/compress.html
4. Uncompress specification https://pubs.opengroup.org/onlinepubs/009695399/utilities/uncompress.html
5. GNU Gzip manual https://www.gnu.org/software/gzip/manual/gzip.html
6. RFC 1951: DEFLATE Compressed Data Format Specification version 1.3 https://tools.ietf.org/html/rfc1951
7. History of Lossless Data Compression Algorithms: The Rise of Deflate https://ethw.org/History_of_Lossless_Data_Compression_Algorithms#The_Rise_of_Deflate
8. bzip2 https://en.wikipedia.org/wiki/Bzip2
9. XZ Utils https://en.wikipedia.org/wiki/XZ_Utils
10. 2.6.38 merge window part 2 https://lwn.net/Articles/423541/
11. zstd https://en.wikipedia.org/wiki/Zstd

Appendix

The table below demonstrates the results of compressing different types of data using tools described in this episode. While not totally rigorous, I did run each compression and decompression multiple times to ensure I was getting consistent results. The laptop I used has an Intel Core i5-6200U CPU running at 2.30GHz, and the system had at least 5 GB of free memory for each run. While this processor has two cores and can run four simultaneous threads, all utilities were run single-threaded.

The term "best" means the highest level of compression available (the exact level used is shown). For bzip2 , the default is the best. For zstd , "best" is -19, which is the highest "normal" level, but "ultra" levels that are even higher also exist. Ratios are the percentage of the original size that the file was reduced to (other sources might instead express the compression ratio as the reduction in size achieved). In all results, smaller numbers are better.

┌────────────────────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────────┬─────────────┐
│                            │    gzip     │    gzip     │    bzip2    │     xz      │     xz      │    zstd     │    zstd     │
│                            │(default -6) │  (best -9)  │    (-9)     │(default -6) │  (best -9)  │(default -3) │ (best -19)  │
├──────────────┬─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Size (ratio) │ 22,036,508  │ 21,891,623  │ 15,795,698  │ 13,487,768  │ 12,938,464  │ 20,454,657  │ 13,709,078  │
│              │             │ (24%)       │ (24%)       │ (17%)       │ (15%)       │ (14%)       │ (23%)       │ (15%)       │
│English Text  ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(90,532,092   │Compression  │ 4.8s        │ 7.6s        │ 8.5s        │ 49.8s       │ 58.8s       │ 0.6s        │ 65.2s       │
│bytes         │time         │             │             │             │             │             │             │             │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Decompression│ 0.7s        │ 0.8s        │ 3.7s        │ 1.2s        │ 1.2s        │ 0.4s        │ 0.4s        │
│              │time         │             │             │             │             │             │             │             │
├──────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Size (ratio) │ 125,291,122 │ 124,189,544 │ 98,016,512  │ 84,882,492  │ 81,954,344  │ 120,604,855 │ 87,298,645  │
│              │             │ (21%)       │ (21%)       │ (17%)       │ (14%)       │ (14%)       │ (20%)       │ (15%)       │
│Source Code   ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(590,008,320  │Compression  │ 22.0s       │ 39.3s       │ 54.8s       │ 241s        │ 298s        │ 3.7s        │ 348s        │
│bytes         │time         │             │             │             │             │             │             │             │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Decompression│ 5.1s        │ 5.1s        │ 20.3s       │ 8.1s        │ 7.8s        │ 2.4s        │ 2.4s        │
│              │time         │             │             │             │             │             │             │             │
├──────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Size (ratio) │ 32,830,905  │ 32,371,241  │ 26,856,579  │ 20,717,288  │ 20,352,880  │ 28,538,810  │ 23,154,582  │
│              │             │ (19%)       │ (19%)       │ (16%)       │ (12%)       │ (12%)       │ (17%)       │ (13%)       │
│Binary Program├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(171,972,264  │Compression  │ 6.4s        │ 22.4s       │ 18.6s       │ 62.2s       │ 67.8s       │ 0.8s        │ 111s        │
│bytes         │time         │             │             │             │             │             │             │             │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Decompression│ 1.5s        │ 1.5s        │ 5.6s        │ 2.3s        │ 2.3s        │ 0.7s        │ 0.7s        │
│              │time         │             │             │             │             │             │             │             │
├──────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Size (ratio) │ 146,397,772 │ 146,397,757 │ 144,485,451 │ 131,950,232 │ 130,926,780 │ 147,154,979 │ 145,703,840 │
│              │             │ (89%)       │ (89%)       │ (88%)       │ (80%)       │ (80%)       │ (90%)       │ (89%)       │
│WAVE Audio    ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│(164,396,302  │Compression  │ 9.2s        │ 9.2s        │ 25.1s       │ 70.4s       │ 97.7s       │ 0.7s        │ 58.3s       │
│bytes         │time         │             │             │             │             │             │             │             │
│uncompressed) ├─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│              │Decompression│ 2.0s        │ 2.0s        │ 13.5s       │ 12.2s       │ 12.1s       │ 0.6s        │ 0.8s        │
│              │time         │             │             │             │             │             │             │             │
├──────────────┴─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┼─────────────┤
│                            │     gzip    │    gzip     │    bzip2    │     xz      │     xz      │    zstd     │    zstd     │
│                            │(default -6) │  (best -9)  │    (-9)     │(default -6) │  (best -9)  │(default -3) │ (best -19)  │
└────────────────────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────────┴─────────────┘

• English text consists of Titles 1 through 10 of the 2020 U.S. Code of Federal Regulations .
• Source code consists of a tar file containing the Linux kernel source, version 4.0.
• Binary program consists of an ELF-format executable of the pandoc application, version 2.17.1.1 found on Debian 12.
• Audio consists of a 24-bit Signed Integer PCM WAVE file with 2 channels at 44.1kHz, about 10:21 in length. For comparison, the audio-specific flac lossless compression utility reduced this file to 97,962,711 bytes (60%) in 2.6 seconds at the default (-5) level and to 97,714,876 bytes (59%) in 5.4 seconds at the highest (-8) level.

Provide feedback on this episode.

Games Mentioned

• Tile Survive - https://tilesurvivegame.com/en
• Monopoly Go - https://apps.apple.com/us/app/monopoly-go/id1621328561
• Arrows Go - https://apps.apple.com/us/app/arrows-go-arrow-puzzles/id6758326278

Apologies, Lee's mic is a bit metallic due to filtering out background noise from a fan

Provide feedback on this episode.

WARNING AI GENERATED NOTES AHEAD YMMW

Here is a summary of the recorded training session regarding Android hacking from Hacker Public Radio, including web references for the main topics discussed.

Overview

The recording features a security consultant performing a live assessment of an Android application. The consultant uses a custom tool suite called "Jamboree" and various other utilities to test a location-sharing and vehicle management app. The session highlights the increasing complexity of mobile app security, specifically dealing with SSL pinning, encrypted traffic, and anti-tampering mechanisms ¹ .

Environment and Tools

The assessment is conducted on a rooted Android emulator. The speaker utilizes several tools to set up the environment and intercept traffic:

• Jamboree : A custom automation tool developed by the speaker over six years to handle rooting, proxy setup, and app installation within minutes ¹ .
• Burp Suite : The primary interception proxy used to analyze traffic between the app and the production server ¹ .
• Frida : Used to bypass anti-root detection and SSL pinning ¹ .
• Ghidra : A decompiler used to analyze the app's code, specifically helpful for patching the Flutter-based application ¹ .
• Android Debug Bridge (ADB) : Used for troubleshooting, debugging, and analyzing logs ( logcat ) to extract user IDs and location data ¹ .

Technical Challenges: SSL Pinning and Flutter

The target application is built using Flutter and implements rigorous security controls, including SSL pinning, which prevents standard Man-in-the-Middle (MitM) attacks. The app's HTTP client ignores system and user-installed certificates, and it does not respect device Wi-Fi proxy settings ¹ .

To overcome this:

1. Traffic Redirection : The speaker uses iptables commands to force all HTTP and HTTPS traffic through the proxy's IP address at the network layer, bypassing the app's proxy ignorance ¹ .
2. Patching with AI : The speaker leverages AI (specifically mentioning Claude and access to "Kuro") to assist in patching the APK. The AI helped navigate Ghidra and generate Python scripts to bypass the app's protections, allowing the modified APK to trust the auditor's certificate ¹ .
3. Frida Scripts : "Frida anti-root SSL pinning" scripts are executed to further mitigate detection mechanisms ¹ .

Key Vulnerabilities Identified

1. Geolocation Spoofing

The consultant successfully spoofed the device's GPS location using emulator settings (e.g., setting the location to Puerto Rico or Costa Rica). The application accepted this falsified location data as valid, indicating a lack of server-side verification for location origin ¹ .

2. Insecure Direct Object Reference (IDOR) / Broken Access Control

The most critical finding involves the app's user tracking feature.

• The consultant discovered that the API allows querying a user's location via a user_id .
• By intercepting traffic and analyzing adb logcat logs, the consultant extracted their own user_id and the user_id of a second test account ¹ .
• While authenticated as one user, the consultant was able to send a request substituting the user_id with the target's ID. The server responded with the target's GPS coordinates. This confirms that an authenticated user can track any other user's real-time location if they possess the target's ID ¹ .
• Proof of concept was created by copying the request as a curl command to demonstrate the exploit ¹ .

3. Potential Information Disclosure

The consultant began testing a feature that allows users to add vehicles by license plate. The concern is that querying a license plate might return excessive PII (Personally Identifiable Information), such as VIN numbers or registration details, beyond what the UI strictly requires (least privilege issue) ¹ .

4. Access Control (Calendar Feature)

The consultant tested whether calendar events could be accessed by switching user_id parameters. This test resulted in a "401 Unauthorized" error, indicating that this specific endpoint had proper access control in place ¹ .

Web References and Resources

Below are references for the main tools and concepts discussed in the training:

• Hacker Public Radio : https://hackerpublicradio.org/
• Burp Suite (Web Security Testing) : https://portswigger.net/burp
• Frida (Dynamic Instrumentation Toolkit) : https://frida.re/
• Ghidra (Software Reverse Engineering) : https://ghidra-sre.org/
• Android Debug Bridge (ADB) : https://developer.android.com/tools/adb
• OWASP Mobile Top 10 : https://owasp.org/www-project-mobile-top-10/
• OWASP Testing for Insecure Direct Object References (IDOR) : https://owasp.org/www-project-web-security-testing-guide/v42/4-Web_Application_Security_Testing/04-Authorization_Testing/04.1-Testing_for_Insecure_Direct_Object_References
• Flutter (UI Toolkit) : https://flutter.dev/

Provide feedback on this episode.

Hi this is your host, Archer72 for Hacker Public Radio.

In this episode I share some of my findings about a problem with the Newsboat naming of the HPR feeds,

which was brought up in comments about my Newsboat show, HPR4424.

hpr4424: How I use Newsboat for Podcasts: comment #6 : download-filename-format for HPR podcasts

Ken already had some findings of his own about the ccdn.php extension in the feed.

hpr4424: comment #10 : Summary of findings

I thought that this might be able to be fixed on an invididual basis, and set out to ask Claude.ai a few questions.

But first, some colaboration from Dave Morriss about a good renaming format. This was definitely more on Dave’s side than mine, but came up with this.

You can tell Dave’s handywork from the short variable names, which stems from his extensive experience on Unix type machines in the University days.

exif-rename-hpr-dave.sh

#!/bin/bash

URL="$(cat /tmp/hpr-url.txt)"
echo "DEBUG URL: $URL" >> /tmp/hpr-debug.log

AUDIO_URL="$(curl -s "$URL" | grep -Eo 'https?://[^"]*\.(ogg|mp3)' | head -1)"
echo "DEBUG AUDIO: $AUDIO_URL" >> /tmp/hpr-debug.log

if [[ -z "$AUDIO_URL" ]]; then
    echo "ERROR: Could not find audio URL from: $URL" >> /tmp/hpr-debug.log
    exit 1
fi

# Changed destination to HPR-queue
DEST=~/podcasts/hub.hackerpublicradio.org/HPR-queue/

# Record files present before download
BEFORE="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"

wget -nc --content-disposition -P "$DEST" "$AUDIO_URL"

cd "$DEST"

# Record filename just downloaded (new file not in BEFORE)
AFTER="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
DOWNLOADED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER"))"
echo "DEBUG DOWNLOADED: $DOWNLOADED" >> /tmp/hpr-debug.log

~/bin/exif-rename-hpr-dave.sh

# Find renamed file — newest file that wasn't in BEFORE
AFTER_RENAME="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
RENAMED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER_RENAME"))"
echo "DEBUG RENAMED: $RENAMED" >> /tmp/hpr-debug.log

if [[ -n "$RENAMED" ]]; then
    echo "\"$AUDIO_URL\" \"$RENAMED\" downloaded" >> ~/.local/share/newsboat/queue
else
    echo "WARN: Could not determine renamed file" >> /tmp/hpr-debug.log
    echo "\"$AUDIO_URL\" \"$DOWNLOADED\" downloaded" >> ~/.local/share/newsboat/queue
fi

At first the question was about something simple. The input was a query on one of the lines from Kevie’s

hpr4398 :: Command line fun: downloading a podcast

Particularly, the section on To get the latest episode of TuxJam

wget curl https://tuxjam.otherside.network/feed/podcast/ | grep -o 'https*://[^"]*ogg' | head -1

Which I re-wrote to:

wget -nc -P ~/podcasts/TuxJam $(curl https://tuxjam.otherside.network/feed/ogg | grep -Eo 'https*://[^"]*ogg' | sort -u | xargs | head -1)

The reason for $() instead of backticks to enclose a command was that the former was being deprecated.

GNU Bash Reference Manual - 3.5.4 Command Substitution

-nc –no-clobber is to not re-download a podcast -P specifies download directory

I went on a different direction than downloading TuxJam and asked to download the last 2 hpr shows, but head -2 did not work as expected. This turned out to be an issue with the placement of xargs joining all URLs and passing them to wget all at once.

Original:

wget -nc --content-disposition -P ~/podcasts/hub.hackerpublicradio.org/HPR-newsboat-test/ $(curl [http://hackerpublicradio.org/hpr\_ogg\_rss.php](http://hackerpublicradio.org/hpr_ogg_rss.php) | grep -Eo 'https\*://\[^"\]\*ogg' | sort -u | xargs | head -2)

New:

curl http://hackerpublicradio.org/hpr_ogg_rss.php | grep -Eo 'https?://[^"]*\.ogg' | sort -u | head -2 | xargs wget -nc --content-disposition -P ~/podcasts/hub.hackerpublicradio.org/HPR-newsboat-test/

Key fixes:

• Removed the premature xargs before head -2 , so head actually limits the list to 2 URLs
• Moved xargs wget ... to the end, after the list is already trimmed
• Cleaned up https* → https? (the original would also match httpssss , etc.)

Now I wanted the downloaded file to go the the queue file, located in ~/.local/share/newsboat/queue

After several iterations in Claude , it was determined that the audio URL was not being expanded by the %u in the download macro.

The solution was to add a /tmp file to hold the actual audio URL

macro d set browser "echo %u > /tmp/hpr-url.txt && ~/bin/download-and-rename-hpr.sh"; open-in-browser ; set browser "your-normal-browser"

A few Claude questions later… After asking to add a function to the macro in order to add the resulting downloaded file to the queue, the file name hpr1234.ogg was being added to the queue instead of the renamed file.

Two things to fix:

1. The renamed file isn’t being found because ls -t runs before the rename has fully settled, or the glob isn’t matching the new filename format (which includes spaces and semicolons)
2. The existing queue entries show the correct format: "url" "path" downloaded — we need to match that, with downloaded status and the full renamed path including extension

Exif rename script

#!/bin/bash
# ~/bin/download-and-rename-hpr.sh

URL="$(cat /tmp/hpr-url.txt)"
echo "DEBUG URL: $URL" >> /tmp/hpr-debug.log

AUDIO_URL="$(curl -s "$URL" | grep -Eo 'https?://[^"]*\.(ogg|mp3)' | head -1)"
echo "DEBUG AUDIO: $AUDIO_URL" >> /tmp/hpr-debug.log

if [[ -z "$AUDIO_URL" ]]; then
    echo "ERROR: Could not find audio URL from: $URL" >> /tmp/hpr-debug.log
    exit 1
fi

DEST=~/podcasts/hub.hackerpublicradio.org/HPR-newsboat-test/

# Record files present before download
BEFORE="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"

wget -nc --content-disposition -P "$DEST" "$AUDIO_URL"

cd "$DEST"

# Record filename just downloaded (new file not in BEFORE)
AFTER="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
DOWNLOADED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER"))"
echo "DEBUG DOWNLOADED: $DOWNLOADED" >> /tmp/hpr-debug.log

~/bin/exif-rename-hpr-dave.sh

# Find renamed file — newest file that wasn't in BEFORE
AFTER_RENAME="$(ls "$DEST"*.{ogg,mp3} 2>/dev/null | sort)"
RENAMED="$(comm -13 <(echo "$BEFORE") <(echo "$AFTER_RENAME"))"
echo "DEBUG RENAMED: $RENAMED" >> /tmp/hpr-debug.log

if [[ -n "$RENAMED" ]]; then
    echo "\"$AUDIO_URL\" \"$RENAMED\" downloaded" >> ~/.local/share/newsboat/queue
else
    echo "WARN: Could not determine renamed file" >> /tmp/hpr-debug.log
    echo "\"$AUDIO_URL\" \"$DOWNLOADED\" downloaded" >> ~/.local/share/newsboat/queue
fi

Using comm -13 on sorted before/after file lists is more reliable than ls -t — it finds exactly which file is new regardless of timing. Check /tmp/hpr-debug.log after the next run to see what DOWNLOADED and RENAMED show.

GNU Coreutils - Comm invocation

Newsboat config snippet

key changes

download-filename-format "%t.%e" 
macro d set browser "echo %u > /tmp/hpr-url.txt && ~/bin/download-and-rename-hpr.sh"; open-in-browser ; set browser lynx

Provide feedback on this episode.

Dave and Kevie continue the Beer Garden series with a look at a relatively new style of IPA known as a Triple. Dave samples Triple Hazy Jane by Brewdog , whilst Kevie tries out Glory Triple IPA by Northern Monk .

Connect with the guys on Untappd:

• Dave
• Kevie

Provide feedback on this episode.

PFSense

https://www.pfsense.org/

Chromebook

https://www.google.com/chromebook/discover-chromebook/

AMD Sempron 140

https://www.techpowerup.com/cpu-specs/sempron-140.c820

Trinity

https://www.trinitydesktop.org/

XFCE

https://www.xfce.org/

Chrome OS

https://chromeos.google/

SSH

https://www.ssh.com/

Onshape

https://www.onshape.com/en/

TinkerCAD

https://www.tinkercad.com/

Thorium OS

https://thorium.rocks/thoriumos

Tech And Coffee

https://techandcoffee.info/

Panera

https://www.panerabread.com/en-us/home.html

IHOP

https://www.panerabread.com/en-us/home.html

Waffle House

https://www.wafflehouse.com/

In And Out Burger

https://www.in-n-out.com/

Economies Of Scale

https://www.investopedia.com/terms/e/economiesofscale.asp

Dunkin Donuts

https://www.dunkindonuts.com/en

F-Droid

https://f-droid.org/en/

Cheap Yellow Display

https://blog.decryption.net.au/posts/cyd-for-beginners.html

NTP Server

https://www.ntppool.org/en/

Beagle (Dog)

https://www.akc.org/dog-breeds/beagle/

Siamese Cat

https://www.life-with-siamese-cats.com/

Bsides InfoSec Conference - Knoxville, TN

https://www.papercall.io/bsides-knoxville-2026

CI/CD Pipeline

https://circleci.com/blog/what-is-a-ci-cd-pipeline/

Strace Command

https://www.geeksforgeeks.org/linux-unix/strace-command-in-linux-with-examples/

High Pass Filter

https://www.izotope.com/en/learn/6-ways-to-use-a-high-pass-filter-when-mixing

Waters & Stanton - Radio Shop

https://www.hamradiostore.co.uk/

Home Assistant

https://www.home-assistant.io/

ESP 32

https://www.espressif.com/en/products/socs/esp32

EMF Camp

https://www.emfcamp.org/

YAML

https://yaml.org/

ChatGPT

https://chatgpt.com/

TOR

https://www.torproject.org/

IPTables

https://linux.die.net/man/8/iptables

https://ccrma.stanford.edu/planetccrma/man/man8/ipchains.8.html

RSYNC

https://linux.die.net/man/1/rsync

SYM Link

https://stackoverflow.com/questions/1951742/how-can-i-symlink-a-file-in-linux

CDN (Content Delivery Network)

https://www.cloudflare.com/learning/cdn/what-is-a-cdn/

Mastadon

https://joinmastodon.org/

DuoLingo

https://www.duolingo.com/

Fedora

https://en.wikipedia.org/wiki/Fedora

Pea Coat

https://www.artofmanliness.com/style/clothing/mans-guide-pea-coat/

Haiku

https://www.haiku-os.org/

Hunt Brothers Pizza

https://www.huntbrotherspizza.com/

Papa Johns Pizza

https://www.papajohns.com/omni/en

PIzza Hut

https://www.pizzahut.com/

Dominos Pizza

https://www.dominos.com/

Marcos Pizza

https://www.marcos.com/

Little Ceasars Pizza

https://littlecaesars.com/en-us/

Hungry Howies Pizza

https://www.hungryhowies.com/

MOD Pizza

https://modpizza.com/

Papa Murphys Pizza

https://www.papamurphys.com/

Wolfman Pizza

https://wolfmanpizza.com/

Fuel Pizza

https://www.fuelpizza.com/

Shallow Hal

https://www.rottentomatoes.com/m/shallow_hal

South Coast Pizza

https://southcoastpizza.com/

Casa Dora

https://www.casadoraitaliancusinepizzeria.com/

National Pizza Day

https://www.nationaldaycalendar.com/national-day/national-pizza-day-february-9

Sagitarius

https://www.zodiacsign.com/zodiac-signs/sagittarius/

Alexa

https://alexa.amazon.com/userProfile?redirectTo=%2F

Gemini

https://gemini.google.com/app

Netscape

https://isp.netscape.com/

Seamonkey

https://www.seamonkey-project.org/

Thunderbird

https://www.thunderbird.net/en-US/

ULC Minister

https://www.ulc.org/

Church Of Spiritual Humanism

https://spiritualhumanism.org/

Oberon Zelle Ravenheart

https://en.wikipedia.org/wiki/Oberon_Zell-Ravenheart

Temple of Wicca

https://www.wiccanfamilytemple.org/

Church Of All Worlds

https://caw.org/

Progressive Universal Life Church

https://www.thepulc.com/

Pine Time

https://pine64.org/devices/pinetime/

AmazFit Watch