Galagic

Moving On

Fri, 15 Nov 2024 15:08:02 +0000

It’s been a while since the last update. This week we’ve moved the website you’re reading from one web host (Gandi) to another (GitLab Pages) and in the process open sourced the code behind it. So if you’re curious about the code itself you can check it out at galagic_com (gitlab.com).

We do need to update the theme on this site as it’s got seriously out of date and I’m not happy with its performance/load. I think we’ll be looking for something a lot simpler as all we need is good old text and images. That’s a job for another day though.

Bigger changes have been afoot but I think I’ll save those for another post. If you do want to know what we’ve been working on take a look at our in-progress educational game: Inner City Learners.

Hacking on a Ledger

Fri, 01 Dec 2023 15:51:14 +0000

Ledger (ledger-cli.org) is a gem of a program for managing your finances. Dating from 2003 it embodies the UNIX philosophy of being highly minimalist and modular. In the form of a command-line tool it lets you run queries on your financial data, held in ledger files that have a simple, textual format. The format is so simple in fact you can easily write them by hand.

The elegant design of Ledger birthed a whole ecosystem and a new approach to personal accounting. For an excellent guide visit Plain Text Accounting (plaintextaccounting.org).

I’ve been using Ledger for a long time but one thing has always caused me a problem. In the UK our tax year starts on the 6th April. It’s useful to organise your financial records around this date, starting and ending each year on that day. Ledger however always considers months, weeks and years to start on the first day of the calendar period. You can change which day Ledger considers a week to start on, but this is not enough to cope with the UK’s tax year.

Why does the UK use such a strange date? Like many things in the UK we should probably blame/thank Henry VIII. For a deeper dive into that see Why the UK tax year begins on April 6 (theconversation.com).

I found some discussion of this problem in Ledger’s forums and repository but it looked like no one had ever gone as far as adding a feature. For a project that had made my life a lot easier I was happy to make a contribution and see if I could fix this myself.

Getting set

Ledger has some good help to get started on development for-the-impatient (github.com). Most usefully it has a big python script acprep that attempts to configure your machine for development. The only drawback was the uncertainty about exactly what it would do to my system. Would the system changes live happily alongside other projects I’m working on? I didn’t want to risk breaking anything so using a development container seemed the obvious choice to isolate any changes.

I am a Fedora user and toolbx (containertoolbx.org) is integrated so well that it’s always what I try first. First I create a new container just for ledger:

> toolbox create -c ledger
> toolbox enter ledger

Next I check out the ledger code and try the big script:

> git clone git@github.com:ledger/ledger.git
> cd ledger
> ./acprep dependencies

It completes without error so I try to build the project:

> ./acprep update
acprep: INFO: Invoking primary phase: update
acprep: INFO: Executing phase: update
acprep: INFO: Executing phase: pull
Already up to date.
acprep: INFO: Executing phase: make
acprep: INFO: Executing phase: config
acprep: INFO: Executing phase: configure
acprep: INFO: System type is => Linux
acprep: INFO: Setting up build flavor => debug
acprep: ERROR: Cannot find CMake, please check your PATH

CMake is missing. I guess this is probably the build tool for the project, but I’m a bit surprised it wasn’t installed by acprep dependencies. I install it via dnf and try the build again. This time I’m missing a C++ compiler. Taking a step back I realise I probably need to start by installing the basic tools. Perhaps the acprep script assumes the compiler etc are already present. So to try and save myself some time I use a dnf groupinstall to try and get a fully working development setup:

> sudo dnf groupinstall "Development Tools" "Development Libraries"

I try the build again - no difference! I am surprised again but perhaps “Development Tools” does not cover C++ tools. Let’s address that:

> sudo dnf install gcc-c++

Later I discover the dnf group “Development Tools” in fact contains things like git, subversion, patching, diffing tools etc but no actual compilers. You can see this with command dnf group info "Development Tools". My mistake. I thought it might contain such things to actually work on the Linux kernel and distro itself.

Anyway back to the build. We get further now but still no success:

> ./acprep update
...
CMake Error at CMakeLists.txt:233 (message):
 Ledger requires GMP version or greater, but only GMP version .. was found

This looks odd. Some kind of complaint about GMP but the message seems to be missing a few details. First I try install to GMP: it takes a bit of hunting around for the right package name but it’s already installed anyway so it can’t be that.

Jumping into CMakeLists.txt at line 233 I can see its doing some kind of work to pull a version number from wherever it finds gmp.h. I find this file in my container and take a look at the contents. It doesn’t have any version information in it but it does have a very useful comment at the top:

 * This gmp.h is a wrapper include file for the original gmp.h, which has been
 * renamed to gmp-<arch>.h...

Eventually I find that I have a file gmp-x86_64.h and I think this is what ledger actually wants to use. A quick hack in CMakeLists.txt and we finally have a successful build!

This problem is very specific to developing with Linux I think. Small changes between how things are packaged on different distros can cause these little road bumps where you might have to hunt around for the right package name or file. It’s probably best to try and use the distro that the authors of Ledger use themselves. In the README.md Ubuntu, Debian and Fedora are all mentioned though so it’s not exactly clear what is the best choice.

As I said earlier I also prefer toolbx for development containers but toolbx defaults to using your host distro. More support for other distros is coming: Fedora Workstation 39 and beyond (gnome.org). I might also explore Distrobox (distrobox.it) in future which I think is more flexible, but you have to do a little more configuration.

Anyway we’ve got Ledger building and the acprep script they provide worked fine once we got the right dependencies installed. After a bit of digging I also find the command to run the tests: ctest . - they all pass. So we’re in a nice place to start poking at the code.

Poking the bear

I have an idea of where I might need to make my changes (times.cc) but initially I just want to run Ledger and get some debug output to see what’s going on.

For compiled languages you of course need to ensure you have run a build before any changes can take effect. Coming recently from a python project the extra delay that compilation causes feels like a real drag on progress, adding minutes to each investigative change I make.

I discover that setting the environment variable LEDGER_DEBUG enables debug output from Ledger which is very flexible. In fact I later find that the debug options are very nicely documented in the Ledger manual: debug options (ledger-cli.org). In addition Ledger already has many useful debug messages included so you can learn a lot just by turning them on.

I add a lot of my own debug print messages but I’m struggling to really get a handle on what’s happening by the time execution reaches the point of calculating time periods. There’s some funky chained function calls/handlers/filters that get setup early and then executed in order to produce reports and it’s not yet clear to me how it all hangs together. See for example the code in chain.cc (github.com).

I realise I need to see the call stack whilst ledger is running to understand what’s happening but have been wary of messing with a debugger. The last time I used one was from within an IDE. It was very easy but I’m using vim and have never tried debugging with it. However there’s no other option if I’m going to make progress.

Before we can debug we need to build the binary with debug symbols. This is as simple as adding a flag to the compiler’s options but I end up hacking this into the build process as I cannot see how to enable it easily through the acprep script I’m using to build. There must be a simpler way I’ve missed. I later discover this is documented in CONTRIBUTING.md (github.com). If only I had the patience to reads the docs first!

So now I’ve got a debug build with symbols (which adds a bit more time to the build unfortunately). With some trepidation I install gdb and open its manual expecting to wrestle with it for some time. Happily I find gdb is actually very approachable and has an excellent manual. I simply prepend my desired ledger command with gdb and I get a command line interface from which I can set breakpoints, step, continue etc with very obvious commands. In no time at all I find I can print the backtrace I wish to see and walk up and down it examining the variables in each frame.

Ledger makes use of the boost library which I understand is pretty standard in C++. I see a lot of boost in the backtraces but largely I can just ignore it and step into the source code before and after it.

The only confusion I suffer with gdb is around some mysterious SIGTTOU signals which keep stopping execution. It initially misleads me into thinking some Ledger code is being called twice (it isn’t) as gdb stops before and after the signals are received on my breakpoints it seems.

It turns out this is a common issue when debugging a process that writes to the terminal. Gdb also writes to the same terminal causing the process under observation to be signalled. The way around this is to remotely debug the process from another terminal. It sounds complicated but in fact you just need to use gdbserver instead of gdb and open another terminal window to connect from. There’s a good summary of the process in the article Debugging binaries invoked from scripts with GDB (redhat.com).

It all feels like hard work. Did I mention that I haven’t touched any C++ code in over two decades?

Making the change

I was finally in a position to be able to make changes, debug and run tests - the full dev cycle. Throughout this I made repeated use of git commit and git stash to make small steps forward/backward as I stumbled towards a better understanding of what was going on.

Most of the time was spent understanding how Ledger sets up report handlers in one context and then runs them later in another with populated data. I repeatedly had to jump into the debugger to check my understanding and print out backtraces. A visual debugger which let me jump forward and backward in execution time would have been useful. I believe gdb can let you move forward and backward in time but when I tried it said it was not supported with the Ledger binary.

I made heavy use of ripgrep to find code references and definitions but ideally my editor (vim) should be setup to do this. I’ve used language servers with other codebases before via the Language Server Protocol (github.io). I need to look into this for C++ though.

Another point of difficulty for me was the use of macros in the source code. Their presence makes debugging just a bit more tricky as the debugger could not map from macro code back to the source files. There are ways around this e.g. expanded C macros (stackoverflow.com) but I could not get anything to work. Probably my inexperience here. I did read a stack overflow answer suggesting macros are best avoided these days but I don’t know how popular that opinion is in the C++ world.

Eventually I worked out the right changes to make and where to put them. I added some tests for my changes and got them passing. Running the full test suite I discovered I’d broken the forecasting feature somehow. A little exception was needed in the code for that and to save some time I learned how to run individual tests with ctest. It seems a little fiddly compared to other test runners - having to specify a start test number, a stop test number, a step, etc: ctest -I (cmake.org).

By this point my source tree is polluted with a lot of build artifacts, test files, temporary files and others bits. Git status gives me two pages of untracked files which makes seeing my real changes difficult. Maybe I’m doing this wrong as I can’t believe people would put up with that. I prefer to have my build outside of the source tree for exactly this reason and to keep a clean separation. There’s probably some way to tell cmake to do that but I don’t know it yet.

Just one failing test remains - a test that runs all the code in Ledger’s documentation to make sure all the examples are correct (which is great). However I haven’t changed anything in there. I do a quick git stash, jump back to master to check the tests were ok before all my changes. They are. I unstash again and run the tests again. They all pass now, weird. Never mind then!

Flagging it up

To finish I add documentation on my changes into the Ledger manual. As I’m writing I realise how weird my change is - it only works on periods of months and years, not weeks. I avoided changing weeks as it made a bunch of existing tests break. Hold on, that doesn’t feel right.

The next day while doing something totally different, the idea to put my changes behind a command line flag pops into my head. If I do that then I can have my change work for all periods and not break existing users and tests. Seems obvious now!

I add a flag to the command line and now I can also put all my tests into their own file, much cleaner. But ctest doesn’t pick up my new test file. Turns out that requires a full clean according to a chat from 2014(!): new tests not being run (groups.google.com). Weird.

All that’s left is just some final tidy-ups to the code to make sure variables have sensible names. I realise I should probably rename the whole feature to ‘align intervals’ as I am starting to understand the jargon that’s used in the manual around time periods. It makes sense. I do a bit more casual testing with various weird intervals but it all seems to work fine. Ok we’re nearly there.

Reaching out

How to approach a project when you are ready to put forward a change? Not all projects make it clear how they like to receive suggestions. The docs suggest raising a bug. I feel this is the best approach, particularly as the issue has been discussed previously and code changes were welcomed at the time. I could have gone to the forums to discuss it first but feel that as I am ready to suggest a code change the repository is best.

I’ve seen projects cheerfully welcome PRs in chat but when you do submit something it gets overlooked. There’s a lot written about maintainer burnout and a PR adds to the workload of a busy team, so I don’t think you should ever expect a reply.

Anyway I get my branch into shape with some git cherry-picking, pulling out the bits that are personal to my setup and push it all up. I create a bug (github.com) and then a pull request (github.com) to resolve it.

Final thoughts

The delays in working with a compiled language are significant. Having just come from a Python project where the change/test/investigate cycle was very fast the difference was stark. I had to be strategic about when to kick off a compile as it took several minutes. The lack of a REPL also stopped me trying things out quickly.

It makes me wonder if there’s a JIT for C++, even just for use during the dev/test cycle to speed things up. I had a quick look and it seems there is a thing called CLing: JIT compiler for C, C++, and the likes (stackexchange.com). I might try that next time.

I had apprehensions at the start, I’ve not touched C++ in decades. I thought it would be a relatively simple change - in the end it was - but the process to get there was not smooth. However I learned a lot and grew some confidence. Particularly when I got gdb working and could really see what was going on inside Ledger.

I hope my change is useful to the project, even if it’s not accepted in its current form but here’s hoping.

Flatpaking

Fri, 03 Nov 2023 13:00:47 +0000

I only use one snap app on my system and I wanted to see if it could be made available as a flatpak as well. The app in question is the OnlyKey-App (github.com), for use with their excellent USB security key: OnlyKey (onlykey.io). The app is written to run on node/Chromium (NW.js). This was my first time building a flatpak and I thought it might be straightforward. It was not.

The flatpak build process is quite particular - builds are performed in a forced offline environment which means a lot of work must go into obtaining online sources/dependencies before the build starts and language tools such as npm must be able to work offline too. I believe this stems from a desire of the flatpak project to achieve reproducible builds in future, though I could not find any official statement of this goal anywhere (see also: Flatpak builds are not reproducible and why that’s a practical problem (ranfdev.com))

Building a flatpak is mostly about crafting a suitable manifest (flatpak.org) that describes how to build and package your app. You can look at the manifest of existing apps (github.com) to see how it works or the handy quick start examples in flatpak-builder-tools (github.com).

That last repo also contains language-specific tools from the flatpak project that are essential for creating a working manifest. Luckily there’s a tool just for node and it’s aware of NW.js too: flatpak-node-generator. Essentially it examines your package.json/package-lock.json and for each dependency it’ll replace it with a local file reference and an instruction to flatpak-builder to download and store the dependency before the offline build step.

Unfortunately I hit two problems with the tool that I had to first overcome:

Npm provider’s generated-sources.json missing many packages (ENOTCACHED) (github.com). My manifest did not include all the packages needed by the app, so the build failed. The fix involved clearing up the way the tool looked through the package-lock.json. I put a PR together for that.

Once this was solved I hit a separate issue - the installer for NW.js does not work offline and of course that is required for flatpak. There was already a bug filed (#77 (github.com)) and it was fixed recently but not yet released. So I had to fork that and patch our app so it uses the fork.

This caused me to run into the second problem with the flatpak-node-generator tool:

NotImplementedError: Git sources in lockfile v2 format are not supported yet (github.com). It’s not possible to use a git reference with the tool whilst on the latest version of node (actually the latest lockfile format). This was also solveable but in the process I couldn’t really ignore all the various ways in which npm can add a git reference to a project. So I ended making quite a few tests to try and cover it all and submitted another PR.

Finally after all this I got the flatpak built and running on my own machine 🎉 The final results are here: OnlyKey-App: Unofficial flatpak build of the OnlyKey app

Unfortunately this result is marred somewhat as it’s teetering on a pile of outstanding PRs and patches. I hope the PRs get accepted in some form and also the upstream OnlyKey project decide to take on the maintenance of the flatpak I’ve created so other users can find it easily.

Overall though this was a enjoyable challenge and I learnt much about building flatpaks.

Figular got corked

Tue, 19 Sep 2023 12:07:25 +0100

The Figular domain is being retired shortly. Existing posts and content have been archived to the Figular wiki and historical links on this site have been updated to point there. The code for Figular will remain available at gitlab.

I’m still in love with the idea of creating diagrams from code and making a universal language in which to do it. However I can no longer see a possible path to making a living from it as an libre/free/open source product, at least in a reasonable timeline.

If you’re looking for something similar to Figular then Mermaid and Mermaid Chart (a new company providing development) look to be going in a similar direction. I just hope the code remains completely open.

I must also mention the Kroki project who generously host a huge list of textual diagramming tools. They are looking for sponsors to continue covering their service.

Figular: style unleashed

Thu, 06 Oct 2022 13:56:20 +0100

Figular has had a recent release and adds lots of new options for styling Figures. The example above shows some of the alternative colours and fonts available with the Circle figure.

To read more see the announcement on Figular.com: style unleashed

Figular: org chart released

Thu, 17 Feb 2022 14:40:28 +0000

Figular has just released a new figure, check it out over on the Figular site at Org Chart. It lets you quickly create an org chart from plain text and you can diagram other hierarchies too.

To see a complex example and read more about it check out the announcement: A new Figure - the Org Chart.

Figular: first prototype available

Fri, 05 Nov 2021 10:15:00 +0100

An early version of Figular is now available and ready for you to explore. Head over to the site to try it.

The release announcement “First Steps” describes what it can do. Your feedback will shape the next version so do get in touch to let us know what you think :-)

Figular: beautiful docs from plain text

Fri, 17 Sep 2021 10:15:00 +0100

I am happy to reveal that the first official Galagic creation will be called ‘Figular’.

Creating beautiful documents can mean fussing with design in Word Processors, drawing and presentation software. What if we describe our content and design entirely in plain text instead?

Find out more about the motivation behind Figular in the blog post over on our sister site: Why Figular

Why does Galagic exist?

Wed, 04 Mar 2020 13:42:00 +0000

There are three things that brought me to the point of starting my own company:

Ideas worth building Over the years I’ve kept a note of interesting ideas but many have only been tangentially related to my day job. Now I am at the point of having the time and skills to build them.
Aligning effort with passion I have been lucky to have roles in the past that have been totally absorbing and I am the type to be consumed by my work. However this leaves little energy for anything else. I realised if I didn’t align my occupation with the ideas I wanted to pursue they would never happen.
Perfect working conditions The most productive times I’ve had were working in small teams that were left alone with a clear goal. However these moments have been short-lived. I want to create a working environment that is always agile.

What kind of software company will Galagic be? Free and sustainable:

Free As the use of computer technology grows in daily life so does the need for the software running it to be open, respectful and available to all. Free/libre and open source software (FLOSS) offers the best way to guarantee this.
Sustainable I was always struck by the idea of sustainable pace from Extreme Programming. This concept now has its equivalent in business: zebras. These are companies that aim for sustainable prosperity over exponential growth.

Privacy Policy

Fri, 17 Jan 2020 17:21:05 +0000

This is version 1.2 of the Privacy Policy, as published 4th March 2021.

In General

We do not track you. We try our best to minimise the data we gather. We use free and open tools whenever possible and we do not share your data with third parties except as compelled by law. Unless prevented from doing so by law or court order we will make a reasonable effort to notify users of any disclosure of their information.

We are not responsible for and do not have control over the privacy practices of third-party sites that appear as links on this site. We encourage users to read the privacy policies of any website visited via links from our website.

No Cookies

We do not use any cookies.

Analytics

We collect a minimum of data to ensure the website functions correctly and to understand how the site is used.

Our only data source are the logs written by apache server (apache.org) as configured by our hosting provider Gandi (gandi.net).

These logs collect information your browser sends when you use our website:

Your IP address
Your ‘referer’ (wikipedia.org) - the address of the web page you visited before our website
Your ‘user-agent string’ (wikipedia.org) - the content varies depending on your browser but typically might contain information on your browser, system, platform and extensions.
The URL you visited on our website
The date and time of your visit

We retain the first three fields (IP address, referer and user-agent) in the logs for 7 days to combat abuse and diagnose technical problems, after that the fields are removed. All data is deleted after 28 days.

Attribution

This policy is derived from:

The Privacy Policy (small-tech.org) of the Small Technology Foundation (small-tech.org) which is licensed under the Creative Commons Attribution-ShareAlike 4.0 International (creativecommons.org).
The Privacy Policy (eff.org) of the Electronic Frontier Foundation (eff.org) which is licensed under the Creative Commons Attribution 3.0 United States License (creativecommons.org).

Licenses

Everything we create is free and open source.

All content on the Galagic website, excluding our trademarks and unless otherwise stated, is licensed under the Creative Commons Attribution-ShareAlike 4.0 International (creativecommons.org).

Feedback

Please contact us with any feedback, data rectification or erasure requests by email: data.protection@galagic.com.

Changes to this policy

If anything in this policy changes the differences will be noted below.

Updated 4th March 2021 to version 1.2. Added provision to inform users if we are compelled to disclose their personal data.
Updated 1st March 2021 to version 1.1. Provided a new email address for data protection requests.