KnightCrawler updated to v2, filename results fixed

The first ElfHosted Stremio Addon was a hacky, untested build of iPromKnight's monster rewrite of the backend components of the original torrentio open-source code.

It served us well from Feb 2024, and was my introduction to the wider StremioAddons community, but the rapid pace of the KnightCrawler devs outpaced our build, and so while fresh builds were prancing around with fancy parsers and speedy Redis caches, we ended up with a monster MongoDB instance (shared by the consumers, and public/private addon instances), which would occasionally eat more than its allocated 4 vCPUs and get into a bad state!

To migrate our hosted instance to the v2 code, we ran a parallel build, imported the 2M torrents we'd scraped/ingested, and re-ran these through KnightCrawler's v2 parsing/ingestion process. Look at how happliy our v2 instance is purring along now!

We cut over to the v2 code a few days ago, and since then we've had some users of the Prowlarr indexer pointing out that the results coming back from the KnightCrawler indexer were...

meh.

Also, the results of the addon in Stremio client itself were a bit too.. minimal - it wasn't possible to see the video/audio codec, for example:

This was a carry-over from the original addon code, and the fancy parsing that v2 does inadvertently truncated the results, such that they weren't as useful anymore. There's an upstream PR which fixes this, and the public instance is already updated.

Now (my preference, anyway) the search results are more detailed and useful:

And the Prowlarr indexer is returning useful results again:

Bugfixes

RealDebrid blocks Hetzner IP(v4) ranges

Just in time for some weekend streaming, Real Debrid performed some maintenance, and prevented download/streaming to non-residential addresses. At the time, this included our Hetzner ranges, as well as popular VPN providers (i.e., PrivateInternetAccess).

To reduce the impact on users, we quickly (and without much testing!) turned on IPv6 addressing in our cluster, and configured Zurg to force an IPv6 connection to Real-Debrid. For now, this has restored service - there are some reports of the original issue being resolved, but testing from our Hetzner ranges shows that IPv4 is still blocked.

So, now we have an IPv6 cluster, and all the fun that comes with it...

IPv6 breaks networking

Shortly after the IPv6 enablement, we started to see sporadic issues with our apps. It turned out that when enabling IPv6, we triggered a bug in Cilium (our CNI) which started allocating duplicate IP ranges to our pods, and generally causing havoc.

After trying various "soft" strategies to fix this, the solution ended up being manually updating the PodCIDR ranges on half of our nodes, which has restored us to the stablity we've been enjoying today!

Coming Soon

TRaSH Guides

@LayeZee has been helping users implement TRaSH Guides custom formats to optimize their media selection for streaming (without transcoding). We added Recyclarr, but it's still a tedious process to build a config file. @LayeZee has been working on a ELI5 guide on the process of manually setting up formats, and @pomnz has suggested we pre-configure the Aars with all the custom formats etc, to make new user setup easier.

Watch this space, and shout out if you have experience with the TRaSH setups and how best to implement / maintain them.

(This weekend's project!)

Today's scoreboard

(The negative trends look bad, but they're likely a response to the April 2024 repricing, rather than a disinterest / negative reflection on our platform)

Summary

Thanks for geeking out with us, and please share these posts with related geeks!