October saw the removal of Ceph from our DE cluster, after it was implicated in the third I/O outage, due to our offsite backup design. The US datacenter got 100% more capacity, we started doing 7-day free trials, weekly videos, and bundling our proxy-streaming Stremio addons to make them more attractive.
I'm happy to announce that this month we started contributing towards the development of Zurg, pushing our open source contributions above $1,200!
To get us started, here are some shiny stats for Oct 2024, followed by a summary of some of the user-facing changes announced this month in the blog (which we moved, during the month)...
The stats below illustrate CPU cores used (not percentage). These stats only cover the DE cluster at present, we're working on cross-cluster metrics aggregation to make this data more useful.
The snapshot was taking an hour after the daily "glowup", and you can see the CPU impact of the rollout reflected in the graph.
CPU load is lower than the previous month, contributing factors being the increased migration of users to the US cluster, and the deprecation of Ceph and the public Annatar instance, which consumed significant resources.
Why Hansel & Gretel?
Bundles are datacenter-agnostic, but nodes are specific to each datacenter, and we needed a way to differentiate US nodes from DE nodes. The fairy tale of Hansel and Gretel originates in Germany
This graph represents memory usage across the entire (DE) cluster. Interestingly, tenant memory usage has is on par with last month, while CPU usage is significantly less.
Other high consumers of RAM:
csi-rclone: used for mounting all rclone-compatible storage mounts, primarily RealDebrid libraries
kube-system: the Kubernetes control plane, including the cilium agents which manage the networking / policy enforcement (currently 11K flows/s across 30 nodes)
traefik: all inbound access to the cluster / services
The current trend seems to indicate that contended nodes (hansels) are constantly ~10% utilized (predominantly by lightweight starter kits and Stremio addons), while the dedicated nodes (gretels) peak at 50% utilization, but also revert to very low utilizatio (2%) when not in use. This is what we'd expect since the dedicated nodes are primarily about reserved capacity.
Last month (Sep 2024)'s for comparison:
Retrospective
Cephless migration
In the first week of October, we suffered our third I/O "freeze", attributed to Ceph and our volsync daily offsite backup processes.
A day later, we published our plan to take the DE cluster "cephless", like the US cluster, relying only on local storage and daily offsite backups, rather than complicated, expensive, and fragile network storage.
It took the remainder of the month to transition all the apps off of Ceph and onto TopoLVM-managed, local NVMe storage. We migrated user apps over about a week's worth of maintenance windows, and then migrated internal backend systems over the following few weeks. The last of the 10Gbps ceph nodes was decomissioned in the last week of October.
A few days later, we received notice of a rolling set of "urgent maintenance" windows planned by Hetzner in Nov/Dec (more details below), which would have necessitated a full shutdown of the DE cluster, if we were still using Ceph!
US cluster
Since last month, the US cluster has grown from 20 to 33 users, and we've added 100% capacity for tenant workloads, plus 200% extra control-plane nodes, for high availability.
Here's what the US cluster looks like today (compare this with the kubectl top output in the CPU/RAM sections above):
root@eagle01:~# k top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
eagle01 475m 5% 10632Mi 33%
yankee01 2327m 29% 20906Mi 65%
yankee02 867m 10% 21798Mi 68%
yankee03 1130m 14% 19992Mi 62%
root@eagle01:~#
We've order another 7 yankees for the US DC in November - we may end up with excess capacity in the short term, but given the Black Friday / Christmas season approaches, and the inevitable delays of ordering hardware, I'd rather be over-prepared than under-prepared!
If you'd like to gauge whether the US cluster would be more suitable for your workloads, visit https://speed.elfhosted.com and perform some comparative tests!
US East Coast DC
We don't have an ETA yet, but our US DC supplier have built out their Pennsylvania (PA) datacenter on the east coast, and we've indicated interest in establishing a presence there. With a PA datacenter on the east coast, and the existing WA datacenter on the west coast, we'd be better able to serve our US Elfies!
Please submit your suggestions for what the DNS suffix and node names should be!
7-day free trials are back
We did an experiment, and offered one-day demo instances for $1. These proved popular, but the feedback we received was that 24 hours is not (at all!) enough time to adequately test an ElfHosted stack.
Since users can now use 7-day free trials to "kick the tyres", we no longer offer unconditional 14-day refunds. The intention is that the trial period is used for testing, and once trial converts to a paid subscription, that subscription is paid for until the renewal period. This reduces support time / overhead, while providing what amounts to a generous ~25% off the first subscription for a "testing phase".
We now have a clear and concise refund policy explaining this.
Pay it Forward
We want to make you so happy with your ElfHosted services, that you’ll tell your friends, as a favor to them (not us), and now’s the perfect time to do so, since free trials are back!
To this end, we’ve redone the referral system (it's a new plugin) – each friend you refer will get $10 credit off a bundle, and (for each paid order) you’ll get $10 credit which will automatically be applied against your subscription.
For details and your own referral codes, just Pay it Forward!
Giveaways
We’ve created a new channel in Discord, to offer giveaways in a fair and automated fashion. @mhdzumair (MediaFusion developer) has been giving away vouchers for a month's AllDebrid subscription, and @layezee has plans to arrange more varied giveways in November.
To enter each giveaway, all you have to do is visit the #elf-giveaways channel and click the “enter giveaway” button!
(The giveaway bot function may switch to a Wordpress plugin, if we can find a suitable one!)
Maintenance window "glows up", shifts by 12h
We shifted our maintenance window by 12 hours, to better suit the growing need for our team to be available after a maintenance period, in case of issues.
We've also no longer calling it a "maintenance window" - that's too gloomy and negative - we now call it, a "glow up" , and it's got its own special channel in #elf-glowup!
$1 tier for non-apps (custom domains)
We adjusted the price on "non-apps" (custom domains, exposed apps) such that in most cases they're 90% more affordable, and "stalled" subscriptions (for the handful which were previously $0/month) no longer use up resources when abandoned, since the store never processed a renewal.
Mounting storage into your pods has long been a complicated and fiddly process, with minor typos causing all pods to get stuck. As a result, storage mounts were priced at $9/month, given their propensity to spawn support tickets.
We've refactored how storage mounts work, such that they're now self-service and fail gracefully. This mean you can setup your own rclone-compatible storage mounts, and even if they fail (because rclone isn't magic), the rest of your services won't be impacted.
For more details, plus an instructional video, see this blog post.
ElfBot does discretionary backups
Due to the Cephless migration, ElfBot is once again able to make on-demand backups of your apps! The backups are stored in a 10Gb volume at /backups, for you to manually transfer offsite.
Wanting a short-form way to track the rapidly-changing debrid ecosystem, we've established https://debrid.news. The idea is to collect debrid-related news in a short but sexy format, and make it available to subscribers as email newsletters.
It's not intended to be Elf-specific, but certainly Elf-related - if you've got an interest blogging / cosplaying Lois Lane / Jimmy Olsen, let me know!
Stremio Proxy-Streaming more affordable
With the recent kerfuffle re RealDebrid’s hair-trigger ban-hammer, we’ve seen an increased interest in the Stremio Addons we host which provide “proxy streaming”, protecting users from inadvertent bans due to WiFi/Cellular/VPN switchover, shared account usage, and random VPN traffic redistribution.
We’ve also had recent feedback from users that the current pricing / options on these addons are (1) too expensive, and (2) too complicated.
So we’ve listened, and have been making some improvements, which you'll find described in details, here.
Our goal is that a user can enjoy the full benefits of an addon with a single, reasonably-priced subscription, but upgrade to add more resources (bandwidth) as required.
Mooar apps
It was a fairly slow month for new apps, given all the infrastructure changes, with the following notable exceptions:
Decluttarr
Decluttarr keeps the radarr & sonarr & lidarr & readarr queue free of stalled / redundant downloads (a common issue in debrid-aarr-land)
Feature overview:
Automatically delete downloads that are stuck downloading metadata (& trigger download from another source)
Automatically delete failed downloads (& trigger download from another source)
Automatically delete downloads belonging to radarr/sonarr/etc. items that have been deleted in the meantime (‘Orphan downloads’)
Automatically delete stalled downloads, after they have been found to be stalled multiple times in a row (& trigger download from another source)
Automatically delete slow downloads, after they have been found to be slow multiple times in a row (& trigger download from another source)
Automatically delete downloads belonging to radarr/sonarr/etc. items that are unmonitored
Automatically delete downloads that failed importing since they are not a format upgrade (i.e. a better version is already present)
ErstazTV
ErsatzTV is beta software for configuring and streaming custom live channels using your media library. The software may be unstable and is under active development.
It can do clever things like create “virtual channels” from your existing Plex media, even inserting (although, why would you want to?) your own custom “commercials / fillers” between scheduled airings!
Unlike the mess which is Plex / Emby / Jellyfin + ThreadFin, Channels DVR is the modern, self-hosted IPTV solution. It’s priced accordingly ($8/month subscription from Channels themselves) – it’s a polished, supported, updated app with multi-platform support. You can use it to consume your (own) IPTV subscription, record live TV, and stream your stored media, either in the web browser or across loads of popular devices.
We’d previously had issues bringing Channels DVR to ElfHosted, because of a peculiarity of their setup, but @BSM has figured it out, and written a detailed and clear guide on setting it up.
Your ElfHosted Channels DVR app subscription is classed as a “complex app”, given the transcoding, CPU, and bandwidth requirements it brings, and you can take advantage of the tiered pricing that bundled users get on all extra apps (Rangers, for example, get 44% off)
ImageMaid
Bundled free with Kometa, ImageMaid is a CLI too clean up the extra overlays and images that Kometa creates, when it does its thing. If you’re a Kometa user, you should see ImageMaid appear on your apps dashboard after the next rollout.
You’ll need to generate and configure a Plex token in order to use it, but it’ll direct you on how to do this
Coming up
Hetzner scheduled maintenance
If you're in the DE cluster (if your app URLs end in .com), then at some point in the next 5 weeks, you'll be impacted by Hetzner's maintenance schedule, for up to 2 hours.
Take a look at this blog post, which lists affected nodes and dates, and includes links to status pages with calendar entries, to keep track of when your apps will be unavailable.
Express checkout
The checkout process in our newly refreshed store is still clunky, and adds more friction than we'd like. Turning on the "next-gen" block-based checkout gives us "express checkout" with Apple/Google Pay, but it breaks ElfBuckz, and the ability for users to pick their own username on checkout.
I've found some documentation on migrating the ElfBuckz (Woocommerce Account Funds) plugin we use for block-based checkout support, so once I've solved this (and the small matter of allowing users to choose their own username on checkout), we should be able to switch over to express checkouts!
Are you a Wordpress dev?
If you've got some expertise in this regard, and you'd like to help out, I'd appreciate any developer input!
Even mooar apps
Apps currently requested can be found (and submitted!) here
Suggestarr is a promising-looking app to automate media content recommendations and download requests based on user activity in media servers like Jellyfin, Plex, and now Emby. It retrieves recently watched content, searches for similar titles using the TMDb API, and sends automated download requests to Jellyseer or Overseer.
Guide / blog refresh
While we've completed the migration of the blog from the docs site to the store, the refreshing of our guides is still on the short-term to-do list.
Now that we have a mostly-workable theme for the store, the ElfVengers and I will be moving much of the guides from the docs site (https://elfhosted.com) to the store, since this is more suitable for frequent, collaborative edits.
We're not sure yet exactly where the division should be, but the docs site should remain technical and price-agnostic, whereas the store should present what we offer, commercially, as simply and attractively as possible.
Torbox Teamup (continues)
Wamy from Torbox is keen to team up with us on a product bundle / offering to support ElfHosted Torbox users. We're still ironing out the details, but rather than trying to use Torbox to replace RD for gigantic-library-building, possible applications are:
Using your private trackers with the Aars for "infinite streaming Plex", while continuing to seed back and maintain your ratio.
Adding a "backup" debrid service for your most critical / popular content, so that you have a fallback in the event of RD rate-limits / changes.
Sourcing non-torrent, premium quality from Torbox's usenet sources.
In typical IT geek fashion, I've slightly widened the scope on this one to capture the creation of alternate packages to RealDebrid, including Torbox, DebridLink, and (hopefully) AllDebrid!
Got ideas for improvements? Send us an EEP (ElfHosted Enhancement Proposal) here!
How to help
Another effective way to help is to drive traffic / organic discovery, by mentioning ElfHosted in appropriate forums such as Reddit's r/plex, r/realdebrid, and r/StremioAddons, which is where much of our target audience is to be found!
I am new here, but today I learned realized that Elfhosted is one of the best free and open source software communities I've seen, and FOSS communities have been at the center of my life since the 90s (Perl, PHP, Symfony, Drupal, Ethereum, etc.). Great open software built by great people who care = great community, and that is something special.
You've done an amazing job @Funky Penguelf with the platform you provide and this place has an awesome mix of active community caretakers and software creators that I've seen here so far like BSM, Spoked, LayeZee and other elf vengers. Keep up the energy, productivity and community and take time to enjoy it and appreciate each other!
I self host and share a fully automated ‘arr stack with Plex. Been doing so for around 4 years. Also recently got into real debrid and hosting a Comet and Annatar for Stremio. The amount of time and head banging I’ve put into it is in the hundreds to thousands of hours. From setting it up to keeping it running smoothly. Let’s not forget the cost of my server and how much it cost to keep it running.
Anyway I wanted to see what ElfHosted was about to compare. Yeah I had the whole thing setup in just a few hours. It also passes the headache of maintaining it to ElfHosted. Will I keep it no because nerdy things and maintaining my server are my hobby and quirky passion project. Will I recommend it to my friends who don’t have the money up front to buy a server, the knowledge to maintain it or desire.
Just my server alone was $2k. Power cost to keep it on yearly is $250ish, annual memberships to RD, Usenet and indexers are around $100. Then whatever a value my free time at. Which is currently at minimum my hourly pay at work or more. Yeah so take the monthly cost of all that and compare to ElfHosted Ultimate Stream package at $39 monthly, add RD to the cost and get nearly all your time back is incredibly cheap.
Lastly it seems like a lot of people forget how quickly an ultimate cable package used to cost. Or how quick paying for every stream service would add up to. Which when using ElfHosted with RD is essentially and more what you get. Quick hint it’s far above the asking price.
As a happy Elfhosted customer—who also self hosts MANY things across about 10 severs (dedicated, VPSes, and VMs running on Synology), I wouldn’t switch to self hosting the services I get from Elfhosted. They just work with very little effort configuring things, and the support the owner and his team provides is second to none. Plus I love being part of a fledgling—but quickly growing—enterprise.
I recently found ElfHosted and decided to start out with the Infinite Starter Kit. Within a week I realized that this was for me and upgraded to the Hobbit plan. Give it another week and I was up to the Ranger plan.
I just love the simplicity and the fact that things just work. For years I've ran a home server and between the constant maintenance and always upgrading harddrives, it became apparent I wanted to make it easier on my self. Enter ElfHosted.
Setup was super easy with the guided documentation and the discord community. It seems that somebody is available at all hours of the day to help with questions. I started with the Aars, which I knew from my prior hosting... but saw a newer product called Riven. I decided to jump in feet first. I enjoy being on the front end of an up and coming replacement for the Aars and will soon be upgrading to the annual plan!
I’ve been using this service for a while now, and honestly, it’s a game-changer compared to anything else I’ve tried for managing my media library. The support is fantastic—super quick, and if the staff aren’t around (which rarely happens), the community steps up right away. I can’t imagine going back to any other platform.
Before this, I had my own setup with a NUC, NAS, and tools like Sonarr and Radarr. It worked pretty well for a while, and my internet speed was high enough to stream without any buffering. But in the end, it wasn’t worth the time or headache of managing all the storage and keeping everything running smoothly.
Now, with this service, everything runs smoothly in 1080p+ with no buffering issues. The interface is really easy to use, which makes managing everything a breeze. Plus, having a whole community of smart people available for guidance is a huge bonus.
I was sold from the start, which is why I quickly upgraded from a 1-month to a 3-month subscription, and I’m planning to switch to a 1-year plan soon. This service totally pays for itself, and I’m sure you won’t be disappointed. It’s been really impressive.
Best possible options for anyone looking for the do-it-all option along with the best customer service ive experienced in this space so far. Id rate it a 6 if I could but its limited to 5/5...
(responding to a Reddit thread re the cost of ElfHosted vs mainstream streaming / self-hosting):
I didn't know that the goal of this project was to compete with large companies running/renting entire DCs. I was under the impression that the goal of this project was to manage the updating of almost selfhosted applications on a shared platform with other users. Basically, be my sysadmin for me.
That being said, paying for services is the 'easy button'. There is a real world cost incurred for the time saved. Time is money. Time is the most valuable currency that exists. Once time is spent, it's forever lost, one cannot retrieve it again (yet). In my mind, there are 3 options for use of time with respect to: mainstream, selfhosting, elfhosted.
mainstream - my time is valuable and I don't want curated content and I don't care what content that I have the ability to consume. I only like what's popular.
elfhosted - my time is valuable, I want my own curated content without being forced to browse past the same damn entry 500 times just to find out that I can't watch the movie I want because it's not available in my current location or was removed last week from mainstream providers.
selfhost - I care about costs and I have nothing but time to waste or I want to learn about the backend of the systems involved. I'll pay for my own VPS/homelab, electricity, manage the OS, manage app updates, figure out how to make the apps talk nice to each other, create my own beautiful frontend.
I know how much my time is worth, does that reddit poster know how much their time is worth? Without knowing what you are worth, you can't make effective capital expenditures with respect to the time it will take to recoup the capital.
I know I don't need elfhosted at all for my use case. I choose to stay with elfhosted because it's my 'easy button'. It's an efficient capital expense for the amount of time it saves me managing my own hardware, apps and saves me electricity costs. I'm also in a situation where I don't have upload bandwidth from my home to serve HD content to myself remotely. If I lived back in a city, I would still be here. My time is worth $$/hr.
"Just wanted to check in here and let @Darth-Penguini and anyone/everyone else know...WOW. I have been struggling with storage for years, maintenance of Docker containers, upkeep, all of it. Elfhosted is so freeing. It's an amazing service that I hope to be a member of for a long, long time!"
"I just have to say, I am an incredibly satisfied customer. I had been collecting my own content for nearly 20 years. Starting off with just a simple external HD before eventually graduating to a seedbox with 100TB of cloud storage attached and fully automated processes with Sonarr and Radarr . However, the time came when the glory days of unlimited Google Drive storage ended. I thought my days of having my full collection at my fingertips via :plex: were behind me, until I found Real-Debrid and ElfHosted.
Now I essentially have the exact same access to content as I had before, but even better. Superior support and community involvement. Content is available almost immediately after being identified. A plethora of tools at my fingertips that give me more control and automation than ever before. Wonderfully well done and impressive! I am looking forward to being a customer for a very long time! Massive kudos to @funkypenguin 🤟
"I would recommend ElfHosted to anyone. It has been great so far and made life a lot easier than running my own setups. If you’re in the fence give them a try and help support this great community."
"thanks for the help and must say this is the best host I every had for my server 🙂 10/10 🙂 All other places I have try have I got a lot buff etc. Your host can even give me full power on a 4K Remux on 200GB big movie file . That's damn awesome 😄"
@tjelite (Discord)
"What an amazing support system these guys have Chris and Layzee i think it was! Both are very patient with me even though I am a newbie at all this. Very thorough and explained everything step by step with me
I couldn’t ask for anything better than the service I have received by these guys! Happy happy client❤️"
"I had my own plex-arrs setup on hetzner for years. Yesterday I deleted everything as elfhosted has gone above and beyond it. And it has a fantastic, active community as well! Very friendly, helpful and like-minded folks always willing to help and improve the system. Top notch!"
"@BSM went above and beyond to make sure I had all the one on one support needed with my sub. Thank you for your patience! Elfhosted continues to be Elftastic !!"
"really enjoying the service from elfhosted. The setup is really easy from the guides on the website. And the help on the discord channel is really quick."
Includes "opportunity cost" of deferring billable consulting work for ElfHosted development! ↩↩
Includes trial subscribers, not all of whom will convert to paid ↩
Some of these must actually be paid yearly in advance, but are represented here monthly for consistency. Confirm my sponsorships here. Includes one-time sponsorships as a result of revenue-sharing, including Riven, Comet, and Zurg↩