pfSense, Cloudflare and ACME - upstream time out error

I have watched Lawrence three YTs about this and also Raid Owles and a few others.
So I managed to set it up once a few months back. During the Christmas-break I wanted to start from scratch. So I removed the ACME package and the certificates.

After this I am not able to create a valid certificate, I get an “broken” button and this message in the log:

2023/01/03 21:03:35 [error] 60761#100105: *4857 upstream timed out (60: Operation timed
 out) while reading response header from upstream, client: 10.1.1.241, server: , request: 
"POST /acme/acme_certificates.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-
fpm.socket", host: "10.1.1.1:8181", referrer: "https://10.1.1.1:8181
/acme/acme_certificates.php"

I have tried everyhting now it feels like and no matter what I do, I get this error.
Please help me, I want to know what I have fxxk up to make this happen over and over.

Are you using DNS API Key? That would be the recommended way.

When I create the certificate I add CFs, Account ID, Zone ID, Global API and CF Mail and the Token which I have created one for the domain as Edit. Should I add the Global API key there to?

I am using Digital Ocean and they have a API key creation process.

Yeah that I know so do Cloudflare, I have created a API key for this which I am using but this seems not to be the problem. When digging into the ACME logs at /tmp/acme, I found that it complains about me missing a TXT _acme-challange record.

TXT for _acme-challenge.domain.io - check that a DNS record exists for this domain

And this error:

server: nginx
date: Wed, 04 Jan 2023 19:09:49 GMT
content-type: application/problem+json
content-length: 144
boulder-requester: 898486957
cache-control: public, max-age=0, no-cache
link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
replay-nonce: Ytp0p-Wk6HsJ_B-nN6jeV-vQ
^M'
[Wed Jan  4 20:09:49 CET 2023] code='400'
[Wed Jan  4 20:09:49 CET 2023] original='{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Unable to update challenge :: authorization must be pending",
  "status": 400
}'
[Wed Jan  4 20:09:49 CET 2023] response='{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Unable to update challenge :: authorization must be pending",

As the error says, it expects a TXT record to exist which is needed if you are doing DNS verfification.

That I spotted myself and tried to add one, as in the image bellow.


I still get a time-out error after adding this.

2023/01/05 14:08:06 [error] 61002#100196: *4924 upstream timed out (60: Operation timed out) while reading response header from upstream, client: 10.1.1.241, server: , request: "POST /acme/acme_certificates.php HTTP/2.0", upstream: "fastcgi://unix:/var/run/php-fpm.socket", host: "10.1.1.1:8081", referrer: "https://10.1.1.1:8081/acme/acme_certificates.php" 

And in the acme issuecert log I found this:

[Thu Jan  5 14:08:24 CET 2023] domain.io:Verify error:Incorrect TXT record

So the question is not, how do I add a correct TXT record?

Are you trying to use the DNS alias mode? If so, I’m not sure how it is supposed to work, because I never used it. But maybe you can read here: DNS alias mode · acmesh-official/acme.sh Wiki · GitHub and try to adapt it for pfSense.

You probably don’t have to, if you choose the correct method for your DNS provider from the dropdown menu in pfSense. Because if you do so, the acme client can log in to your DNS provider via API, and create the record automatically…

Another possibility would be to issue or renew the certificate manually, you can do so by switching to the “DNS manual” method in pfSense and then click on the “Issue” button. After that the content for the TXT record will be displayed in the WebUI. It will be a randomly generated key which is only valid once. Looks something like this: “kldjasu89dsf9w83ruhjiwi8UHHHJkjéIo8i”`

Btw. I would recommend you to switch the “Acme Account” to “staging” while you are playing around, Otherwise you will hit the rate limit of Let’s Encrypt sooner or later…

I studied the issuecert log some more and I saw that it succeeds with the certificate, but on another of my cloudflare domains. I had a certificate for this domain months ago, but I have removed them from pfSense all together before I began with this.

So how do I go forward from this?

@bb77 Good catch with the staging part! :slight_smile:

I would say all you have to do is choose one of the three methods here https://github.com/acmesh-official/acme.sh/wiki/dnsapi#1-cloudflare-option, and then fill out the corresponding fields here:

If this doesn’t work, I’m afraid I can’t help you any further. Maybe the pfSense forums, Google or someone else here still has a few ideas…

As I wrote above it to get an certificate for a domain that is not configured under the ACME add-on. I have configured it, months ago, and I did remove it from the ACME add-on and the Cert Manager. Still the config files are present in /tmp/acme/.
I have posted in the Netgate forums asking if I safely can remove all files for those I know I don’t have in the Acme add-on.

Second, I thought all those values had to be populated, so I did.

So I removed the zone ID, account ID and Token, thus only using email and Global API Key.
The GUI says it fails but in the log I find this:

[Thu Jan  5 16:31:07 CET 2023] original='{
  "status": "valid",
  "expires": "2023-01-12T15:27:49Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "domain.io"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/1928367"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/898957/15708087",
  "certificate": "https://acme-v02.api.letsencrypt.org/acme/cert/03f21f21767972ea44bbc"
}'
[Thu Jan  5 16:31:07 CET 2023] _json_decode
[Thu Jan  5 16:31:07 CET 2023] _j_str='{
  "status": "valid",
  "expires": "2023-01-12T15:27:49Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "domain.io"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/1928367"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/89957/1570087",
  "certificate": "https://acme-v02.api.letsencrypt.org/acme/cert/03f262d70b8d1767972ea44bbc"
}'

That has to say that it succeeded right?

What exactly does it say?

Hmm, not sure, but this doesn’t look like a new certificate. If it were newly issued by Let’s Encrypt, it would expire on 2023-04-05…

There is an GUI glitch as you say, a reload of that page said all is swell!