Help with setting subdomains with a VPS IP for Headscale & NTFY

Hello all,

A year ago I got my own VPS and configured Headscale. Didn’t setup any domain with it and just used the VPS IP, which works completely fine.

Now I want to setup a subdomain with the domain I got from the same VPS provider so I can use Headscale with my subdomain, setup my DERP server, etc.

The provider has a setting in the panel that allows me to automatically set a hostname for the VPS by providing a FQDN. I don’t know if that’s what I should do first and then manually also setup a subdomain and point to the same VPS IP, or not set the FQDN at all and only configure an A record for a subdomain with the VPS address to then use with the Headscale configuration.

Also because I never used a domain with a VPS and only used just the IP, I’m not sure if I need to change anything in the /etc/hosts file or leave it as is since I’m not setting the VPS hostname to the domain I own.

As mentioned on the title, I also plan to setup NTFY, which is going to need it’s own subdomain pointed to the same VPS IP because I want to self host that too.

I’m on Debian 12, any help and resources share are appreciated. Here is a rundown of what I want to do.

I bought a domain with the same VPS provider and my VPS is Debian 12. I don’t use the domain with the VPS or anything else at the moment. I want to:

  • Configure a subdomain with my already configured Headscale
  • Install and also configure NTFY with a subdomain

Thanks in advance.

I don’t use either of those but it should be fine just to creat the DNS A records with your provider and not worry about setting up anything special on the host itself unless those services call for it in their setup.

DNS Configuration

headscale.example.com   A   <Your VPS IP>
ntfy.example.com        A   <Your VPS IP>

Hello Tom, thank you for the reply. I’ll try just that then, I thought I had to also change files in the system besides the appropriate configuration with Headscale.

Out of curiosity, do you have any VPS instances for projects in general and do you use any specific notification system for system notification delivery?

This forum runs on a VPS as does my web site. I have https://www.netdata.cloud/ setup on both to let me know if there are any issues.

Hey Tom, I did this and set my subdomain as per your video but I now get this when I do systemctl status headscale.

http: TLS handshake error from 145.224.119.62:29358: acme/autocert: missing server name

Should I add 127.0.0.1 sub.example.com in my /etc/hosts possibly or something like that?

I also don’t know if it’s my configuration with the domain I did. there are extra preconfigured entries such as CNAMEs, MX records, NS. Shoud I remove everything and only keep the DNS A record with host headscale.example.com pointing to my VPS IP?

I would make sure the FQDN is resolving properly on the server running Headscale.

Update: Solved the problems.

For the domain, the problem was that there were CNAME wildcards that I never seen on another registrar and all I had to do was delete everything and only keep a single DNS A record for my subdomain. Also because the previous changes where still cached, I was checking via dig very quickly and was getting the old records. Waited a bit and then finally started seeing the A record correctly pointing to my VPS IP.

For the acme/autocert: missing server name error, the problem was not a Headscale configuration. The errors were coming from the already registered nodes in my tailnet because I now use a domain and port 443 instead of IP and port 8080. All I had to do was re-register the nodes with the new domain and port 443.

1 Like

Hello again Tom, don’t know if I should ask here or create a new post for someone to answer but I will just ask here just in case.

Now that I have Headscale all setup with a DNS A record of headscale.mydomain.com pointing to my VPS IP that is using ports 80,443, will the other subdomain pointing to the same VPS IP have any problems?

Headscale uses port 80 with autocert for LE/ACME certificate renewals (with HTTP-01 Challenge).

I will setup a ntfy.mydomain.com DNS A record to point to my VPS IP, and will use Certbot to create a certificate (with DNS Challenge) that will serve NTFY to a port such as 4444 for example. I know how to do all that now.

My question is, if both subdomains point to the same VPS IP and each service uses a different port, will the host/VPS know where to direct that traffic, or I will need to configure UFW to forward traffic for example? I also watched your NGINX Proxy Manager video, but don’t know if that’s applicable to begin with and if I could skip all that.

Any information appreciated. Let me know if I should create a new post for someone else to reply!

The ports respond to the request, no the DNS name. So if the ports are different then it does not matter. but If you want a service to use the same ports then you will need a proxy (such as NGINX Proxy Manager) that can use the SNI information to send it to the proper service behind the proxy.

I understand.

So basically since Headscale will be on port 443 and NTFY on port 4444, there is no problem since they will use 2 different ports?

Thank you for the reply!