"ElfReport" for Feb 2024
We're 7-months old now, starting to outgrow our (1Gbps) baby 
clothes, and exploring a new service niche, the hosted Stremio addons!
February saw us outgrowing our 1Gbps storage nodes, and becoming more involved in the r/StremioAddons community, moving from hosting a single hacky alternative for the popular-but-overloaded torrentio service, to providing free / subscription hosting for many popular Stremio Addons.
To get us started, here are some shiny stats for Jan 2024, followed by a summary of some of the user-facing changes announced this month in the blog...
Stats
Resources
Most apps consume almost no CPU while idle - the larger consumers are streamers doing transcoding, and download clients doing download/unpack operations.
CPU usage roughly roughly doubled from last month (the big dip at 06:00 was a planned shutdown for maintenance)
Based on a quick metrics snapshot, we're more contended for RAM than we are for CPU (elves and giants run tenant workloads):
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
dwarf01 228m 2% 23405Mi 73%
dwarf02 181m 2% 22535Mi 70%
dwarf03 222m 2% 24886Mi 78%
dwarf04 350m 4% 24256Mi 76%
dwarf05 202m 2% 24290Mi 76%
dwarf06 171m 2% 24145Mi 75%
dwarf07 169m 2% 23619Mi 74%
dwarf08 259m 3% 24938Mi 78%
dwarf09 461m 5% 24577Mi 77%
dwarf10 178m 2% 24767Mi 77%
elf01 1353m 8% 47069Mi 36%
elf02 2045m 12% 42965Mi 33%
elf03 1699m 10% 48951Mi 38%
elf04 1404m 8% 34240Mi 26%
elf05 2344m 14% 40551Mi 31%
elf06 1818m 11% 37628Mi 29%
elf07 2238m 13% 40636Mi 31%
elf08 1225m 7% 41491Mi 32%
elf09 1389m 8% 39357Mi 30%
elf10 1384m 8% 47221Mi 36%
elf11 1461m 9% 49015Mi 38%
elf12 1480m 9% 60994Mi 47%
elf13 1421m 8% 43242Mi 33%
elf14 1265m 7% 39742Mi 30%
elf15 1742m 10% 39376Mi 30%
elf16 2862m 17% 44473Mi 34%
elf17 1322m 8% 43008Mi 33%
elf18 1427m 8% 36301Mi 28%
elf19 1178m 7% 39423Mi 30%
elf20 1246m 7% 35472Mi 27%
fairy01 2550m 31% 51195Mi 79%
fairy02 2009m 25% 46481Mi 72%
fairy03 1401m 17% 41114Mi 64%
giant01 1781m 14% 23401Mi 36%
giant02 1721m 14% 21396Mi 33%
giant03 2856m 23% 34566Mi 53%
gnome03 815m 10% 17817Mi 27%
goblin04 3128m 26% 61464Mi 47%
goblin05 661m 5% 58512Mi 45%
goblin06 424m 3% 58338Mi 45%
Last month (Jan 2024)'s for comparison:
This graph represents memory usage across the entire cluster. By far the largest consumers of RAM is rook-ceph, since ceph will basically take all the RAM you give it, for caching / performance.
RAM usage has increased by about 50% since the previous month, although some of this may be due to the addition of 10Gbps ceph nodes for SSD-backed storage (the new "tenants" distinction on the graph below should make this more apparent next month)
kubectl top nodes
NAME CPU(cores) CPU% MEMORY(bytes) MEMORY%
dwarf01 228m 2% 23405Mi 73%
dwarf02 181m 2% 22535Mi 70%
dwarf03 222m 2% 24886Mi 78%
dwarf04 350m 4% 24256Mi 76%
dwarf05 202m 2% 24290Mi 76%
dwarf06 171m 2% 24145Mi 75%
dwarf07 169m 2% 23619Mi 74%
dwarf08 259m 3% 24938Mi 78%
dwarf09 461m 5% 24577Mi 77%
dwarf10 178m 2% 24767Mi 77%
elf01 1353m 8% 47069Mi 36%
elf02 2045m 12% 42965Mi 33%
elf03 1699m 10% 48951Mi 38%
elf04 1404m 8% 34240Mi 26%
elf05 2344m 14% 40551Mi 31%
elf06 1818m 11% 37628Mi 29%
elf07 2238m 13% 40636Mi 31%
elf08 1225m 7% 41491Mi 32%
elf09 1389m 8% 39357Mi 30%
elf10 1384m 8% 47221Mi 36%
elf11 1461m 9% 49015Mi 38%
elf12 1480m 9% 60994Mi 47%
elf13 1421m 8% 43242Mi 33%
elf14 1265m 7% 39742Mi 30%
elf15 1742m 10% 39376Mi 30%
elf16 2862m 17% 44473Mi 34%
elf17 1322m 8% 43008Mi 33%
elf18 1427m 8% 36301Mi 28%
elf19 1178m 7% 39423Mi 30%
elf20 1246m 7% 35472Mi 27%
fairy01 2550m 31% 51195Mi 79%
fairy02 2009m 25% 46481Mi 72%
fairy03 1401m 17% 41114Mi 64%
giant01 1781m 14% 23401Mi 36%
giant02 1721m 14% 21396Mi 33%
giant03 2856m 23% 34566Mi 53%
gnome03 815m 10% 17817Mi 27%
goblin04 3128m 26% 61464Mi 47%
goblin05 661m 5% 58512Mi 45%
goblin06 424m 3% 58338Mi 45%
Last month (Jan 2024)'s for comparison:
I'm not sure these stats are accurate, they've likely overly high because pods on the host network (like metallb, ceph, etc) will end up counting all traffic on each host, rather than the pod itself. This is exacerbated with more and more storage nodes, which run in `hostNetwork`` mode. However, the graph still gives a good indication of network usage compared to the previous month.
These samples are fairly useless for trending, since they're taken over a 1h period, because I haven't worked out how to do an efficient Prometheus query of all the metrics required to graph a longer period!
Last month (Jan 2024)'s for comparison:
These are the traffic stats for egress from Hetzner. They exclude any traffic to/from Hetzner Storageboxes, and they also exclude stats from any nodes which were decomissioned during the period, which for February was a lot (we standardized on 128GB i9-9900K nodes). Thus, these stats are likely under-reported for February.
We also wouldn't see the resulting network traffic to ElfStorage or Storageboxes though, since this traffic is not counted as being "external".
Ingress / Egress stats for Feb 2024
Last month (Jan 2024)'s for comparison:
Ingress / Egress stats for Jan 2024
Ceph provides our own storage ("[ElfStorage][elfstorage]"), typically used for long-term slow storage and seeding, as well as all the storage for our per-app /config volumes, now backed by 10Gbps nodes with NVMe disks.
Ceph storage growth has slowed from 100% down to 20% during February, as we've shifted our focus / subscriber base towards "Infinite" Debrid/symlink-based storage. Interestingly, keeping track of thousands of symlinks per-tenant still consumes a significant amount of I/O resources, so although storage capacity is not under pressure at the moment, scaling Ceph was a priority during Feb.
Last month (Jan 2024)'s for comparison:
What's new?
Stremio Addons
The big highlight this month has been our hosted Stremio Addons, which started with a flakey fork of Torrentio, and has now expanded to 8 popular addons, all of which are available publicaly for free, or privately for more customization or rate-limits.
| Name | What does it do? | Subscribe |
|---|---|---|
| AIOStreams | Combine the results from multiple addons for consistent formatting / display, and add proxystreaming via MediaFlow-Proxy | Subscribe! |
| Comet | Searches for content from Prowlarr / Jackett, streams via RealDebrid. Optionally proxies streams, allowing you to use your RealDebrid account from multiple IPs simultaneously | Subscribe! |
| Jackettio | Get torrents from your private trackers using Jackett and delivered via Real-Debrid, AllDebrid, or Debrid-Link | Subscribe! |
| MediaFusion | Especially popular for watching Tamil, Hindi, Malayalam, Kannada, English, and dubbed movies & series. A highly-configurable and capable addon | Subscribe! |
| NuvioStreams | A powerful Stremio addon that provides direct HTTP streaming links for movies and TV shows from multiple online providers | Subscribe! |
| WebStreamr | Simple Addon which provides HTTP streams (no Debrid needed) for popular content in multiple languages  | Subscribe! |
| YourIPTV | Watch your paid IPTV (user/pass required) subscriptions in Stremio | Subscribe! |
AllDebrid support
Real-Debrid gets all the attention, but there are a lot of AllDebrid users who were left out in the cold, because AllDebrid blocks API access to "server IPs", which include our Hetzner infrastructure. We've architected a bolt-on VPN for KnightCrawler, RDTClient and our rclone mounts, so that AllDebrid users can access their media as "first-class citizens" again!
VPN provided by ElfHosted
BYO VPN
Currently the following are supported with Private Internet Access, but we can add more VPNs upon request:
Symlinks Supercharged
The "Advanced Infinite Streaming" bundles utilize symlinks to make it appear to Plex / Jellyfin or Emby that your debrid library is locally available.
How to help
If you'd like to make a donation in recognition of our infrastructure costs, our open-source resources, or our friendly support, a simple donation product is available at https://store.elfhosted.com/product/elf-love/ 
Another effective way to help is to drive traffic / organic discovery, by mentioning ElfHosted in appropriate forums such as Reddit's r/selfhosted, r/seedboxes, r/realdebrid, and r/StremioAddons, which is where much of our target audience is to be found!
What's coming up?
Regular pricing re-balancing starts 1 April 2024
Just like we re-balance our node workloads daily to meet resourcing / responsiveness targets, we're going to need to start regularly adjusting our pricing to meet budget / break-even targets! After 7 months of production operation, we have enough data, and a wide enough pool of users, that we should be able to reasonably estimate how much each app costs to run, such that the platform can become self-sustainable (the current app prices were guesses without adequate data to back them up).
Look for updates on this pricing balancing idea during March, with the intention to apply our first changes from 1 April - I'll be be posting updates and requests for input in the next week or two. If you have any feedback / suggestions / experience in this regard, let me know!
Backlog management
There's an ever-growing list of app bugs, suggestions, improvements, and opportunities that are gathering dust in my Things.app inbox. Recently several community members have offered their time / expertise to help with outstanding tasks.
To better surface all these tasks, I'll be moving them to GitHub issues - this will not only help us to track the backlog and deal with bugs etc, but make it easy for fellow elves to contribute suggestions, improvements, and expertise.
Plex library migrations
We've started to run into I/O contention on the big, busy storage volumes, which use CephFS to make files available to multiple pods (for example, to Plex for storage, but to FileBrowser for browsing config). In early March (already started as I write this), we'll be migrating users' Plex libraries (some of which are >50GB!) to Ceph "block" storage, which bypasses the I/O contention, at the cost of being unable to "view" your Plex metadata / DB / logs via FileBrowser (we'll come up with an alternate solution as required).
PostgreSQL support for the Aars
Radarr, Sonarr, etc all now have built-in support for PostgreSQL as a replacement to the troublesome and easy-to-corrupt SQLite database they come with by default. To support our KnightCrawler database, I've started using CloudNativePG (CNPG) for full-lifecycle database management. With a simple CR, CNPG will establish a highly-available PostgreSQL cluster, including automated failover, incremental and automatic backups to local snapshots and to an S3 bucket.
This declarative approach to creating a PostgreSQL database could allow us to, in bulk, create individual database for Arr instances, such that setup is still fully automatic, and users just "get" the PostgreSQL-enabled instances. I'm testing this internally, so look out for updates during March!
Offering free demos
Nothing gets my attention on a new app like a live demo. I've considered reaching out to open source projects who don't have their own online demo, and offering to host a self-resetting demo for them.
This approach has been successful with the Stremio Addons, and I think it'd be effective at gaining exposure to more niche communities.
A hosted demo would provide their potential users the opportunity to evaluate the app "live", and would also drive more traffic / recognition / SEO juice towards ElfHosted.
If you've got an open-source, self-hosted app and you'd like a free demo instance hosted, hit me up!
(We are currently donating torrent hosting to https://freerainbowtables.com)
Your ideas?
Got ideas for improvements? I'd love to hear them, post them in #elf-suggestions!
Join us!
Want to get involved?
Want to get involved? Join us in Discord and come and test-in-production!
-
For simplicity's sake, we're only presenting stats for https://elfhosted.com here, and not https://store.elfhosted.com. Fully transparent traffic details are available for elfhosted.com and store.elfhosted.com ↩↩
-
All moneyz are in US dollarz ↩
-
Includes "opportunity cost" of deferring billable consulting work for ElfHosted development! ↩
-
Cluster costs are calculated based on Hetzner montly invoicing, which lags behind by a month. So this month's changes are only reflected in next month's invoice! ↩
-
These are actually paid yearly in advance, but represented here monthly for consistency. Confirm my sponsorships here ↩