Announcing hz.tools

Ever since 2019, I’ve been learning about how radios work, and trying to learn about using them “the hard way” – by writing as much of the stack as is practical (for some value of practical) myself. I wrote my first “Hello World” in 2018, which was a simple FM radio player, which used librtlsdr to read in an IQ stream, did some filtering, and played the real valued audio stream via pulseaudio. Over 4 years this has slowly grown through persistence, lots of questions to too many friends to thank (although I will try), and the eternal patience of my wife hearing about radios nonstop – for years – into a number of Go repos that can do quite a bit, and support a handful of radios.

I’ve resisted making the repos public not out of embarrassment or a desire to keep secrets, but rather, an attempt to keep myself free of any maintenance obligations to users – so that I could freely break my own API, add and remove API surface as I saw fit. The worst case was to have this project feel like work, and I can’t imagine that will happen if I feel frustrated by PRs that are “getting ahead of me” – solving problems I didn’t yet know about, or bugs I didn’t understand the fix for.

As my rate of changes to the most central dependencies has slowed, i’ve begun to entertain the idea of publishing them. After a bit of back and forth, I’ve decided it’s time to make a number of them public, and to start working on them in the open, as I’ve built up a bit of knowledge in the space, and I and feel confident that the repo doesn’t contain overt lies. That’s not to say it doesn’t contain lies, but those lies are likely hidden and lurking in the dark. Beware.

That being said, it shouldn’t be a surprise to say I’ve not published everything yet – for the same reasons as above. I plan to open repos as the rate of changes slows and I understand the problems the library solves well enough – or if the project “dead ends” and I’ve stopped learning.

Intention behind hz.tools

It’s my sincere hope that my repos help to make Software Defined Radio (SDR) code a bit easier to understand, and serves as an understandable framework to learn with. It’s a large codebase, but one that is possible to sit down and understand because, well, it was written by a single person. Frankly, I’m also not productive enough in my free time in the middle of the night and on weekends and holidays to create a codebase that’s too large to understand, I hope!

I remain wary of this project turning into work, so my goal is to be very upfront about my boundaries, and the limits of what classes of contributions i’m interested in seeing.

Here’s some goals of open sourcing these repos:

• I do want this library to be used to learn with. Please go through it all and use it to learn about radios and how software can control them!
• I am interested in bugs if there’s a problem you discover. Such bugs are likely a great chance for me to fix something I’ve misunderstood or typoed.
• I am interested in PRs fixing bugs you find. I may need a bit of a back and forth to fully understand the problem if I do not understand the bug and fix yet. I hope you may have some grace if it’s taking a long time.

Here’s a list of some anti-goals of open sourcing these repos.

• I do not want this library to become a critical dependency of an important project, since I do not have the time to deal with the maintenance burden. Putting me in that position is going to make me very uncomfortable.
• I am not interested in feature requests, the features have grown as I’ve hit problems, I’m not interested in building or maintaining features for features sake. The API surface should be exposed enough to allow others to experiment with such things out-of-tree.
• I’m not interested in clever code replacing clear code without a very compelling reason.
• I use GNU/Linux (specifically Debian ), and from time-to-time I’ve made sure that my code runs on OpenBSD too. Platforms beyond that will likely not be supported at the expense of either of those two. I’ll take fixes for bugs that fix a problem on another platform, but not damage the code to work around issues / lack of features on other platforms (like Windows).

I’m not saying all this to be a jerk, I do it to make sure I can continue on my journey to learn about how radios work without my full time job becoming maintaining a radio framework single-handedly for other people to use – even if it means I need to close PRs or bugs without merging it or fixing the issue.

With all that out of the way, I’m very happy to announce that the repos are now public under github.com/hztools.

Should you use this?

Probably not. The intent here is not to provide a general purpose Go SDR framework for everyone to build on, although I am keenly aware it looks and feels like it, since that what it is to me. This is a learning project, so for any use beyond joining me in learning should use something like GNU Radio or a similar framework that has a community behind it.

In fact, I suspect most contributors ought to be contributing to GNU Radio, and not this project. If I can encourage people to do so, contribute to GNU Radio! Nothing makes me happier than seeing GNU Radio continue to be the go-to, and well supported. Consider donating to GNU Radio!

hz.tools/rf - Frequency types

The hz.tools/rf library contains the abstract concept of frequency, and some very basic helpers to interact with frequency ranges (such as helpers to deal with frequency ranges, or frequency range math) as well as frequencies and some very basic conversions (to meters, etc) and parsers (to parse values like 10MHz). This ensures that all the hz.tools libraries have a shared understanding of Frequencies, a standard way of representing ranges of Frequencies, and the ability to handle the IO boundary with things like CLI arguments, JSON or YAML.

The git repo can be found at github.com/hztools/go-rf, and is importable as hz.tools/rf.

    // Parse a frequency using hz.tools/rf.ParseHz, and print it to stdout.
	freq := rf.MustParseHz("-10kHz")

	fmt.Printf("Frequency: %s\n", freq+rf.MHz)
    // Prints: 'Frequency: 990kHz'

    // Return the Intersection between two RF ranges, and print
    // it to stdout.
	r1 := rf.Range{rf.KHz, rf.MHz}
	r2 := rf.Range{rf.Hz(10), rf.KHz * 100}

	fmt.Printf("Range: %s\n", r1.Intersection(r2))
    // Prints: Range: 1000Hz->100kHz

These can be used to represent tons of things - ranges can be used for things like the tunable range of an SDR, the bandpass of a filter or the frequencies that correspond to a bin of an FFT, while frequencies can be used for things such as frequency offsets or the tuned center frequency.

hz.tools/sdr - SDR I/O and IQ Types

This… is the big one. This library represents the majority of the shared types and bindings, and is likely the most useful place to look at when learning about the IO boundary between a program and an SDR.

The git repo can be found at github.com/hztools/go-sdr, and is importable as hz.tools/sdr.

This library is designed to look (and in some cases, mirror) the Go io idioms so that this library feels as idiomatic as it can, so that Go builtins interact with IQ in a way that’s possible to reason about, and to avoid reinventing the wheel by designing new API surface. While some of the API looks (and is even called) the same thing as a similar function in io, the implementation is usually a lot more naive, and may have unexpected sharp edges such as concurrency issues or performance problems.

The following IQ types are implemented using the sdr.Samples interface. The hz.tools/sdr package contains helpers for conversion between types, and some basic manipulation of IQ streams.

The following SDRs have implemented drivers in-tree.

The following major packages and subpackages exist at the time of writing:

hz.tools/fftw - hz.tools/sdr/fft implementation

The hz.tools/fftw package contains bindings to libfftw3 to implement the hz.tools/sdr/fft.Planner type to transform between the time and frequency domain.

The git repo can be found at github.com/hztools/go-fftw, and is importable as hz.tools/fftw.

This is the default throughout most of my codebase, although that default is only expressed at the “leaf” package – libraries should not be hardcoding the use of this library in favor of taking an fft.Planner, unless it’s used as part of testing. There are a bunch of ways to do an FFT out there, things like clFFT or a pure-go FFT implementation could be plugged in depending on what’s being solved for.

hz.tools/{fm,am} - analog audio demodulation and modulation

The hz.tools/fm and hz.tools/am packages contain demodulators for AM analog radio, and FM analog radio. This code is a bit old, so it has a lot of room for cleanup, but it’ll do a very basic demodulation of IQ to audio.

The git repos can be found at github.com/hztools/go-fm and github.com/hztools/go-am, and are importable as hz.tools/fm and hz.tools/am.

As a bonus, the hz.tools/fm package also contains a modulator, which has been tested “on the air” and with some of my handheld radios. This code is a bit old, since the hz.tools/fm code is effectively the first IQ processing code I’d ever written, but it still runs and I run it from time to time.

    // Basic sketch for playing FM radio using a reader stream from
    // an SDR or other IQ stream.

    bandwidth := 150*rf.KHz
    reader, err = stream.ConvertReader(reader, sdr.SampleFormatC64)
    if err != nil {
        ...
    }
    demod, err := fm.Demodulate(reader, fm.DemodulatorConfig{
        Deviation:       bandwidth / 2,
        Downsample:      8, // some value here depending on sample rate
        Planner:         fftw.Plan,
    })
    if err != nil {
        ...
    }
    speaker, err := pulseaudio.NewWriter(pulseaudio.Config{
        Format:     pulseaudio.SampleFormatFloat32NE,
        Rate:       demod.SampleRate(),
        AppName:    "rf",
        StreamName: "fm",
        Channels:   1,
        SinkName:   "",
    })
    if err != nil {
        ...
    }

    buf := make([]float32, 1024*64)
    for {
        i, err := demod.Read(buf)
        if err != nil {
            ...
        }
        if i == 0 {
            panic("...")
        }
        if err := speaker.Write(buf[:i]); err != nil {
            ...
        }
    }

hz.tools/rfcap - byte serialization for IQ data

The hz.tools/rfcap package is the reference implementation of the rfcap “spec”, and is how I store IQ captures locally, and how I send them across a byte boundary.

The git repo can be found at github.com/hztools/go-rfcap, and is importable as hz.tools/rfcap.

If you’re interested in storing IQ in a way others can use, the better approach is to use SigMF – rfcap exists for cases like using UNIX pipes to move IQ around, through APIs, or when I send IQ data through an OS socket, to ensure the sample format (and other metadata) is communicated with it.

rfcap has a number of limitations, for instance, it can not express a change in frequency or sample rate during the capture, since the header is fixed at the beginning of the file.