Spent way too much time hunting down a strange ActiveMQ issue recently. Finding the cause was complicated by several factors:
The ActiveMQ servers are managed by an external party
Our application runs on Kubernetes, also managed by an external party
The problem only occurred on an environment that’s in use by a customer, not in any of our test environments or during development
We connect to the ActiveMQ servers in several different ways, from various Spring Boot applications, sometimes using annotation magic/garbage and Spring Data connection strings, other times using connection factories in Java
Strangely, the problem only occurred in 1 of our microservices, not in any of the others
Firewall rules make it very hard to talk to the ActiveMQ servers from a local machine
Whats the problem?
Application logs were littered with this ominous exception. Without any further context or details.
java.lang.Exception: Failed to find a store at /path/to/broker.ks
at org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.validateStoreURL(SSLSupport.java:399)
at org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeystore(SSLSupport.java:339)
at org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeyManagerFactory(SSLSupport.java:375)
at org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.loadKeyManagers(SSLSupport.java:355)
at org.apache.activemq.artemis.core.remoting.impl.ssl.SSLSupport.createContext(SSLSupport.java:222)
at org.apache.activemq.artemis.core.remoting.impl.ssl.DefaultSSLContextFactory.getSSLContext(DefaultSSLContextFactory.java:50)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector.loadJdkSslEngine(NettyConnector.java:786)
at org.apache.activemq.artemis.core.remoting.impl.netty.NettyConnector$1.initChannel(NettyConnector.java:690)
at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:416)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
A little over a year ago, I was fired from my job. I haven’t really talked about this in public, but in order to celebrate this dubious anniversary, I figured I’d write a blog post about it. Dates and other specifics have been altered slightly, to make it a bit harder to identify the company and people in question.
Back in 2012, I joined a tiny startup. And when I say tiny, I do mean tiny. Two founders (one an engineer/CTO, the other a salesman/CEO), and one part time freelance engineer. When I joined, they had built a prototype, and had managed to attract a first customer. I was asked to help turn the prototype into a product. This was a boatload of fun, even if it was a bit hectic. After all, Mister Sales had over-promised, and we couldn’t really under-deliver to our first customer. This, for better and for worse, became a running theme.
I’m an avid user of Markdown related tools. This blog is written in Markdown. My notes are in Markdown (using Obsidian). I often archive web pages as Markdown. The benefits are many: it’s plain text, easily searchable, easy to read on any kind of device, there’s great tooling out there, and it’s flexible enough to fit most needs.
It is for these reasons that I spent some time creating a tool, RSSDown, which takes an RSS or ATOM feed as input, and outputs Markdown. This makes it easy to read blog posts on the command line. It’s also able to persist the output as Markdown files, which can be useful for archiving or synchronization purposes.
The word “tool” is probably a bit generous here. It’s basically a combination of Flexmark, for Markdown conversion, and Rome, for feed parsing.
I’ve been managing servers in one capacity or another since 1999 or so. A lot has changed since then. Many changes have been gradual improvements. Hardware slowly got cheaper, colo slowly got cheaper, bandwidth got a lot cheaper really quickly, partially managed servers and VPSes have become dirt cheap.
With all those improvements on the hardware side of infra, it’s easy to overlook the improvements on the software side. Thanks to tools like Ansible and Let’s Encrypt, I can now deploy an HTTPS site on new hardware in minutes. Where most of those minutes is spent downloading software updates and rebooting into a new kernel. This may sound silly and trivial in 2022. 20 years ago, this would have taken DAYS, and would have involved any number of manual steps. Not only would it have cost me a lot more time, it would have also cost a lot more money.
So cheers. Remind me to buy the Ansible & Let’s Encrypt folks a beer or two. They’ve certainly earned it.
More fun with Apache HTTPD. This time I wanted to access a user’s email address and user id, both stored in LDAP, in an application without LDAP integration. Thankfully, this turned out to be pretty easy. But first, some background.
The application is a very simple Java application, which sits behind HTTPD as a Reverse Proxy. The Reverse Proxy handles authentication, because reinveting the wheel gets a bit dull after a while, and mod_authnz_ldap works just fine. But while adding a feature to the application, I suddenly needed access to the user’s userid and email address, for Reasons™.
The documentation turned out to be somewhat helpful, if cryptic. There is a relatively easy way to expose LDAP attributes to CGI scripts using mod_authnz_ldap — which can set environment variables for perusal by the CGI script. Using the same mechanism in combination with a reverse proxy was a bit trickier; after all, the Java application is running in a Jetty container and it talks to the proxy using HTTP. That doesn’t really offer room for injecting environment variables.
Input validation has been something of an obsession of mine ever since I picked up PHP back in the 90s. You can never trust user input. At all. To give an example, back when contact forms (without CAPTCHAs) were a thing that existed, many of them were abused to send spam to random people. How? By abusing naïve input validation in combination with PHP’s extremely shitty mail() header implementation. Instead of accepting headers as an array of headers – and only as an array of headers; it accepts them as a CRLF deliminated list of strings. So if you have an input field with the sender’s email address, and you want to helpfully include this in a From header in your email, you’d better validate the thing properly … because if not, spammers will abuse that From address to include CC or BCC headers and they will peddle their prick enhancing pills using your mail server. Or so it used to go, PHP’s implementation has improved somewhat, and developers aren’t as dumb as we were back then.
Ahem. I got a bit sidetracked there. Input validation. Important, yes.
Living in Belgium can have pretty interesting consequences. The puny country has 3 official languages (Dutch, French & German). Many more languages are spoken, of course, this is the 21st century and there are people of every possible origin here, and most countries are pretty similar in that regard. What’s less common is Belgium’s language divide. The country is split into multiple regions, with each region having a single official language — I’m conveniently ignoring Brussels here. This made some sense historically, after all, people had to fight (quite literally) for the right to speak their language. At least in the Dutch part of the country. French was the lingua franca for long time, and the tiny German speaking bit is the result of a couple of wars and is irrelevant in this discussion.
Sometimes you just want to deploy a tool and protect it from unauthorized access. This is easy if the tool comes with built-in access control. Especially if there’s support for LDAP authentication or SAML. But sometimes, that’s just not the case. In which case you can just Apache HTTPD’s built-in mod_auth to enforce authentication with Basic Authentication. This works really well if the tool is something like a static HTML page, or even a PHP application. It gets a bit trickier if the tool requires running in its own process, like maybe a Java application running in an embedded Jetty instance. In which case you can use mod_proxy and mod_auth together. And hey presto, you’re done, right? Wrong.