Host it yourself

[maru]

Way back at the inception of this board we talked a bit, in a very freeform way, about workflows and tools. It came up again talking about computers, just figuring out new ways to organise piles of stuff across computers.

Another example: I moved from streaming services to just having all my music as files again. But now I have lots of computers and mobile devices, so it became a bit unwieldy considering how to keep it all present and current. I used to sync everything with Syncthing, but I kept finding myself losing music outright; a media player would edit ID3 tags, and then a conflict would occur, and Syncthing would conclude CONTAINMENT METHOD BRAVO was needed and nuke the site from orbit.

I would always make this smart playlist on all my devices, "things I haven't listened to yet," where if play count was 0, a song would be added. This meant on every single device I had 24 days of music that just ... waited for me to go through it all again. And seeing as I am extremely something, I kept going through that smart playlist as the main access point for my music. It felt like chipping away at a huge effort. But the effort would restart; get a new computer or just switch computers and it didn't match the way I remembered the effort.

In the time since this board began dear friend @meri put me onto just hosting my music instead with navidrome. And I ended up starting to use it, and things like Immich, on my Mac mini at home. But then I also realised, what if the external HDDs I have, the ones without SMART checks -- what if they just died? Well, photos and music matter more than just, like, stray things to watch or listen to. Even though I back up all these databases, I don't even want to cause the stress on these drives to get to that place. I decided I was better off segregating some stuff to a VPS.

So music and photos now live in a Tailnet VPS, but I still host bookmarks, my RSS feeds and my personal booru at home.

After looking at bep-san here I'm realising I don't host any websites or forums or whatever on my own hardware -- this forum and my fediverse instance and the like are all in a proper data centre, but I could definitely afford to host something from my house with a Cloudflare tunnel, or I guess a public Tailscale DNS entry, though I hate their subdomains. I would rather assign something proper, aesthetic ... something cafe...

I think I could do more, probably? I was considering importing all my mail in one big archive I serve to myself, and remove any storage requirements I would have from an IMAP provider. Even just ssh-ing into my server to notmuch it would be possible. But is this crazy? Who needs to look through their mail that way? Ignoring that services like protonmail have terrible search, making it hard to live from your inbox ...

I wanted to know what you do across your personal network. Graphs and diagrams welcome.

[8sumint]

one day i will write a universe of good software that will solve inter-device synchronization and extensible, relational metadata in distributed and heartwarming ways. until then i know nothing will be good enough to truly satisfy me so i just don't bother trying...

[sinku]

So in the past few days I took my old Thinkpad T430 and set it up with slackware 15 with the goal to host a website. So far, I've set up the box, SSH and port forwarding. So as I type this where I am sitting in Spain, I can SSH into the machine in England. I think that is super cool. I spent time with @8sumint yesterday setting up talkd and using talk, which is this primitive realtime chat program where you can see your friend type out her messages in real time. wall and write are also very cool. It is very cute and instead of a modern chat program where it is horizontal, the layout is vertical, so its almost like a game of tennis.

Anyway, I set this box up to stand up a simple static website. I signed up for a free domain from eu.org (no relation to the european union) and I'm using hurricane electric for DNS, also free. eu.org apparently take a very long time to dispense requested domains. It could take a month or more and they dont even notify you, according to reddit and other forums. I was concerned that they were dead, but apparently not as of may last year. I will probably end up following a guide that @maru gave to me and set up my site with darkhttpd. I keep thinking of what else I could possibly do with this box. AS I was typing this, little things popped into my mind. Maybe I could put a few widgets like a guestbook or a user counter onto my site. Maybe put the free-use ticker on the website. I dunno. But then it'd be out of the purview of darkhttpd and I would need to use something else. Decisions... I already downloaded nginx, but who needs that.

[hint_of_jasmine]

I think you’ve hit on a key thing with storage durability. That was my primary concern — the access software comes and goes (and honestly, given a few years, is something I could make a decent pass at); the data is irreplacable.

The core of my setup is Mumei, a trusty former gaming rig running proxmox. The bedrock, though, is a truenas instance, leyline. It’s got the four drives passed through so it runs a 0+1 zraid for one-drive redundancy and weekly backups to backblaze b2. Backblaze is probably more key here, and if you don’t have 30TiB of data it’s likely a lot cheaper. This offers SMB and NFS to everything else.

For local services, I’ve got the arrs but honestly Infuse over SMB and deluge get more play. This ensures I have a local media collection. I’m currently using PlexAmp for as long as they continue to honour the lifetime subscription I bought in college. Immich for videos. Watch-tracking by the service isn’t important to me since I use MAL and letterboxd, but Infused apparently has a sync for this? And I’m using youtube premium but am considering moving to invidious for sponsorguard.

Access is accomplished through tailscale. I should likely be using my router’s innate WireGuard support, but it was easier to install a tailscale subnet router onto mumei. 6to4 tunneling is important to avoid overlapping IPv4 subnets here, especially from public wifi.

Overall I’m pretty happy with it! And old desktop towers are great at just having a bunch of disks and truenas on them.

[maru]

Immich for videos

Immich? For like, your personal video collection? Do you mean Infuse? Or do you have an intriguing workflow...

[hint_of_jasmine]

Sorry, I meant Immich for photos and Infused for videos Fairly standard setup!

[watermoon]

you know, every so often i've considered running something public-facing off my home network. but for as much as the idea of hosting personal items off my personal hardware appeals to me, the idea of having it traceable back to my home ip spooks me just enough to not bother going through with it.
maybe tailscale could help me overcome that personal worry, though… maybe…

so instead i've just turned to hosting some services (mostly for media) off my nas and tying a domain to it. though then the challenge is that, for as much as my life revolves around organizing things, keeping an entirely separate organization within these apps is a bit too large of a time investment for me to feel ok justifying it if the only person who cares is myself. still, i figure i minds well post about what i'm running and the degree to which i've given up on keeping their libraries looking pretty.

jellyfin: for videos – anime, movies, concerts, music videos. at the start i really did want to make it into a pretty library (do you know how long it took for me to get it to recognize the 2006 anime REC!?) but after a while it became just too much upkeep for something that i don't use as my main way of watching videos.

besides, the collection on there is incomplete anyway, as i've yet to figure out something good to use for my youtube video collection. tube archivist seemed promising, but it assumes that the videos will be stored in a specific human-unfriendly file structure and that you only plan on using it to ever watch your videos, which i see as a pretty big strike against it from a preservation perspective.

navidrome: for my entire music collection. for a while i had this one just as a supplemental way of accessing my music, but after jumping on the feishin train recently, i'm heavily considering just making it my main music player, and so lately i've been moving my metadata over to it.

mealie: for recipes. this one is actually pretty neat and i honestly thought i was going to get more use out of it, but since my wife does the majority of the meal planning and has been using a google sheet to organize her recipes for years, it'd feel wrong of me to drag her over to my own system. i do still try to port some of the recipes in her sheet over though, every so often, just in case…

kavita: ostensibly for books, but i haven't even bothered trying to set this one up. i feel like organizing the mass of ebooks and journal articles i've amassed over the years is bound to be hell any which way, and doing so is pretty low-priority…

qbittorrent: my cute little seedbox, gluetun'd to mullvad.

the lounge: an irc client. back when paralogue irc was on espernet, i set up a whole almalinux vn to run (mostly) soju, and that was such a pain to configure that i ended up scrapping it all and searching for something easier, and this has been quite easy to use. i'm a little tempted to swap this for weechat just so i can use a terminal client while on desktop, but i also don't use irc enough to justify making the switch.

silverbullet: another notebook that i was using as a project notebook for a little while before just folding those notes back into the 11,000-note obsidian slurry. i have opinions about silverbullet – so much so that i was drafting a pretty big post about them a while back before realizing that they could all be summed up by saying that it's the most cowboy-coded program that i think i've ever used.

vaultwarden: for passwords. i'd been using bitwarden for years but decided to switch over to self-hosting it for most of my passwords while using keepassxc for the rest and for redundancy. is this idiotic? did i fall too far down the paranoia hole?

backups?: tbh i just have restic dump copies of everything important onto the nas + backblaze b2. i think i've only needed to rely on it to recover something once so far, though if i were ever in a situation where i lost everything here and had to rely on cloud backups, i think i have a solid 50/50 chance of remembering the password i put on it.

[sinku]

So, update:

My new website lives at https://iruyz.fr.eu.org/ , using darkhttpd on slackware 15. I have had to deal with a variety of asspains like powercuts knocking everything over and needing to reconnect to wifi, shuffle around dns, update IPs because my darn ISP doesn't give me an option for a static ip (GRRR) and futzing around with SSL taking way longer than it should've. But it's live, I have a good workflow to send stuff to the site, a nice style, a homepage with fun buttons that I could really use more of, and soon some little pieces of writing, I think.

[meri]

in 2022 i started collecting music after many years of dealing with the increasingly frustrating streaming landscape. as i amassed more music standardization and cataloguing was automated with picard (where i also started contributing data to the musicbrainz universe when releases were missing) to get my stuff tagged how i want it. that was the start of my hoarding journey...

my dad had a NAS with xbmc when i grew up, and its kind of been in the back of my head for a long time to do something similar. (freud?) i ran a terramaster D5-310 for a year and a half, but i pretty quickly outgrew it because i realized that i wanted apps and not just data storage. the shitty cpu wasn't going to cut it (and also smb is like, horribly slow over wifi, even to just stream music. and you can't take it outside your home without compromises) i built a custom NAS this year with a motherboard SOC which fits nicely inside a jonsbo N2 white case. i definitely over-provisioned a little but i enjoy the extra headroom. i was able to put this entire thing together for under $600 (excl drives) and i suspect it will last me many years.

i run truenas as my software. it can do docker apps and manage the storage file system. i have plenty to say about the pros and cons of the truenas software. but that's a story for another day. i would love to try another system at some point, but that's a massive effort, as you'll soon learn by the stupid amount of things i host on it.

most importantly, my nas is named "inai". you know, like "it's not".

now to talk about the actual hosting of stuff...

music stack (slskd + picard + navidrome + feishin + narjo)

i went through a majority of players and setups, but eventually i settled on running navidrome to have streaming access to my stuff on the go. they have a web ui, which is fine in a rush. feishin is the best desktop client i've found. on ios, i ran plexamp for the longest time as there hasn't been a navidrome client that fit my picky criteria. just recently i've been trying out narjo, a very early (but promising) ios client without even as much as a web presence. i found it on reddit (help) and they have a testflight running. i've been very happy with it so far. it does everything i want more or less, and the progressive offline caching is seriously genius. it's like everything i would've done if i had the skills/time to make something similar.

my typical workflow these days to get new stuff is i run the soulseek client slskd on inai via a web ui, i tag it using picard (also running through a web client) and then navidrome picks it up. i have my tags set up to be multi-valued, so i get multiple artists tagged on each album/track which is a nice bonus.

i expose my navidrome server using cloudflared, which lets me put a domain name to it and you get SSL. hopefully that's secure enough. i don't port forward anything directly.

movies, tv, anime (infuse + plex + *arr + syncthing + transmission)

i have a whole download automation system. the *arr family of apps can negotiate with torrent trackers (prowlarr), manage downloads (tv - radarr, movies - sonarr) and have those files moved off a seedbox using syncthing. the typical workflow when i want something is i add it on sonarr/radarr and i hit the search button and it'll find a suitable release and put it in a folder where plex can find it. when new anime comes out, its RSS feed parser is able to parse that a new anime episode is available, and download it almost like magic. it's shocking that any of this works as well as it does.

i don't like torrenting on my network, so i use a seedbox to do that. transmission can move any downloaded files directly to inai.

infuse is my usual end client app, for the sole reason that it can natively read ASS subtitles on the apple TV without needing to be transcoded

misc stuff

• i run adguard as a DNS server on my network to block tracking websites on the dns level. it also serves an extra purpose to allow multicast for my services via nginx proxy manager (like navidrome.inai.local instead of inai.local:30040)
• i use homepage to provide a directory of services hosted on the nas (accessible on inai.local)
• i use dozzle to view logs of all docker containers
• i host a calibre server to manage books
• (experiment) using photoprism to quickly view and manage camera raw photos?
• i back things up to backblaze using truenas's backup features. i don't save any of the media, because that would be expensive. just the stuff that sucks to lose. pictures, music. anything else can't easily be rebuilt. it's a nice extra bonus on top of zfs allowing me to have one drive fail.

i'm still trying to de-servicify where i can. it's become a bit of an addiction. the only things i have online now are photos (icloud), web hosting, and the things i host which actually need to be up, like milkmedicine. i can't contend with the promise a VPS has to be up most of the time. i break things on my NAS way too much to use it for something that's not just plain entertainment.

[ejtwnt]

i used to host a lot of stuff from my house, from email to media servers… the only thing that’s survived until now is the media server (old pc with hard drives stuffed into it) i keep under my desk, with anything else moved to either a fleet of VPS’s i don’t actually use or services with better uptime than my local power grid.
my plan is to someday get my act together and actually use the servers im paying for, but ill have to get over my terminal NIH syndrome first… no time to rewrite the universe

[maru]

Curious, @meri -- what pushed you over the edge with streaming? Nothing frustrated me but I sort of felt like I was developing a really weird relationship with music where it was pure aesthetic commodity. I started to feel that way about all media, too, and began getting really selective. But now I have way too much stuff to visit...

just recently i've been trying out narjo, a very early (but promising) ios client without even as much as a web presence. i found it on reddit (help) and they have a testflight running. i've been very happy with it so far. it does everything i want more or less, and the progressive offline caching is seriously genius. it's like everything i would've done if i had the skills/time to make something similar.

Yeah, I like that it's picking up the progressive caching since Symfonium does it fantastically (you set up the rules for what it should automatically slurp up, so I just had it auto-download albums and songs I mark as favourites; but then it would cache stuff in playlists or albums that were queued up). Narjo is okay? I keep hitting bugs that make me distrust it, but each day it gets a little better; eg. at first if I downloaded an album it would show up in my Downloads as UUIDs without any info. Then the next day that was fixed.

I balance between Amperfy and Narjo but I suspect I may end up moving fully over eventually.

the only things i have online now are photos (icloud)

If you have these urges I'm surprised you haven't gone to Immich, it's quite solid. I only really needed to do it when I left iOS, but now that I'm back, I understand why I would be less pushed to do so; iCloud Photos feels pretty good by itself. Immich feels similar in UX, though? It is marginally more annoying on iOS if only because Tailscale drains more battery on iOS than on any of my other devices, so I need to be more careful with when I turn it on and off.

@ejtwnt what got you doing email, specifically? That's the one thing I'd be too terrified to do.

[ejtwnt]

@ejtwnt what got you doing email, specifically? That's the one thing I'd be too terrified to do.

it was just in service of doing everything myself. i used a cheap vps from digitalocean as the smtp server and proxy for everything else i hosted though, as i didnt want to expose my home ip address to the internet. i had a lot of stuff hosted, like nextcloud and various caldav servers i tried, as part of my attempt to reduce dependency on big tech companies.

eventually i had my fill and bounced around various email services like mxroute and proton, until i settled on tuta for now.

i think the only things i still actually host on a server I control is the rss feed reader i use (miniflux <3), soju for irc, and jellyfin on the aforementioned pc-turned-media server.
how sad.. maybe ill spin up a gitea/forgejo instance or something just for myself. my website could use a few updates, considering it’s a nearly blank white screen right now.

[meri]

Curious, @meri -- what pushed you over the edge with streaming? Nothing frustrated me but I sort of felt like I was developing a really weird relationship with music where it was pure aesthetic commodity.

losing access to things i liked due to the platform having some kind of licensing problem was what pushed me to it. and over time i learned to be more intentional with my media, and i feel as a result i think i learned to have more respect for the arts...

If you have these urges I'm surprised you haven't gone to Immich, it's quite solid. I only really needed to do it when I left iOS, but now that I'm back, I understand why I would be less pushed to do so; iCloud Photos feels pretty good by itself.

immich and photoprism seem to do the same thing broadly... i was hoping photoprism could do my bidding but immich definitely is a bit more shiny. i'm not sure if it handles camera raws well? need to look into it more. but yeah in the interim icloud has been very good for my iphone's camera roll (which isn't just photos. it's somewhat of a download folder (different use case?)).