Project

General

Profile

Actions

Bug #13696

open

Rebuild Config deletes PTR records but does not add them

Added by Roger Spencer about 8 years ago. Updated about 1 year ago.

Status:
New
Priority:
Normal
Assignee:
-
Category:
DNS
Target version:
-
Difficulty:
Triaged:
No
Fixed in Releases:
Found in Releases:

Description

Select a host in the all hosts section and choosing "Rebuild Config" fails every time with DNS failure. Foreman and Proxy logs show the deletion of the PTR and A records but only shows the A record being added after. Unsure why Foreman reports an error because it doesn't log anything about an error.

Using dns_nsupdate_gss on the proxy. Both Foreman and Proxy are at version 1.10.1. DNS is AD hosted. Proxy running on CentOS 6 & 7 (tried both OSs).

Logs (identifying data masked/changed)

Proxy log

I, [2016-02-13T16:14:11.437756 #6156]  INFO -- : Requesting credentials for Kerberos principal serviceaccount@DOMAIN.NET using keytab /etc/foreman-proxy/dns.keytab
I, [2016-02-13T16:14:11.549343 #6156]  INFO -- : 172.30.XX.XX - - [13/Feb/2016 16:14:11] "DELETE /som-deploy1.humedica.net HTTP/1.1" 200 - 0.1123

I, [2016-02-13T16:14:11.599534 #6156]  INFO -- : Requesting credentials for Kerberos principal serviceaccount@DOMAIN.NET using keytab /etc/foreman-proxy/dns.keytab
I, [2016-02-13T16:14:11.707506 #6156]  INFO -- : 172.30.XX.XX - - [13/Feb/2016 16:14:11] "DELETE /101.68.30.172.in-addr.arpa HTTP/1.1" 200 - 0.1085

I, [2016-02-13T16:14:11.797363 #6156]  INFO -- : Requesting credentials for Kerberos principal serviceaccount@DOMAIN.NET using keytab /etc/foreman-proxy/dns.keytab
I, [2016-02-13T16:14:11.885362 #6156]  INFO -- : 172.30.XX.XX - - [13/Feb/2016 16:14:11] "POST / HTTP/1.1" 200 - 0.0891

Foreman Log

 | Started POST "/hosts/submit_rebuild_config?host_ids%5B%5D=618" for 172.XX.X.XX at 2016-02-13 16:14:09 -0500
 2016-02-13 16:14:09 [app] [I] Processing by HostsController#submit_rebuild_config as HTML
 2016-02-13 16:14:09 [app] [I]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"lrJWHLkduis0PMavAKLJDxU3Rohwu/XbAIdXXZ5N25s=", "host_ids"=>["618"]}
 2016-02-13 16:14:09 [app] [I] Delete DHCP reservation for servername.humedica.net-99:60:56:b0:46:9d/172.XX.XX.XXX
 2016-02-13 16:14:10 [app] [I] Create DHCP reservation for servername.humedica.net-99:60:56:b0:46:9d/172.XX.XX.XXX
 2016-02-13 16:14:11 [app] [I] Delete the DNS A record for servername.humedica.net/172.XX.XX.XXX
 2016-02-13 16:14:11 [app] [I] Delete the DNS PTR record for 172.XX.XX.XXX/servername.humedica.net
 2016-02-13 16:14:11 [app] [I] Add DNS A record for servername.humedica.net/172.XX.XX.XXX
 2016-02-13 16:14:11 [app] [I] Redirected to https://foreman.domain.net/hosts
 2016-02-13 16:14:11 [app] [I] Completed 302 Found in 2858ms (ActiveRecord: 2.9ms)
 2016-02-13 16:14:12 [app] [I]

Changing host information in the Edit Host section works as expected. Changing IP or host name (or both) and both A and PTR records are updated properly.


Related issues 2 (2 open0 closed)

Related to Foreman - Bug #2668: Foreman appears to be incorrectly checking the local resolvers rather than SOANew06/14/2013Actions
Related to Smart Proxy - Support #35087: DNS POST: how to force record creationNewActions
Actions #1

Updated by Dominic Cleal about 8 years ago

  • Category set to DNS

I think this might be the effect of #2668, where Foreman only uses the caching resolver to check for PTR records, but it uses the SOA for A records.

The rebuild will check if the record's already there before trying to create it, so it might see a cached record (even after deletion) and skip the step.

Actions #2

Updated by Dominic Cleal about 8 years ago

  • Related to Bug #2668: Foreman appears to be incorrectly checking the local resolvers rather than SOA added
Actions #3

Updated by Alex Fisher over 7 years ago

I've also found that sometimes A records aren't created either. The problem for me seems to be that the smart proxy updates DNS via one DNS server, but the changes aren't replicated (via AD) to all the other DNS servers foreman has in its resolv.conf until several seconds to a few minutes later.

Reording the nameserver entries in resolv.conf has worked so far. I'm not sure what happens if DNS is temporarily down on the 'primary' server though. The resolver tries the DNS servers in the order they appear in the resolv.conf, but after it moves on to using the 2nd or 3rd entry does it ever switch back to using the first again?

Actions #4

Updated by Alexander Olofsson over 6 years ago

Dominic Cleal wrote:

I think this might be the effect of #2668, where Foreman only uses the caching resolver to check for PTR records, but it uses the SOA for A records.

The rebuild will check if the record's already there before trying to create it, so it might see a cached record (even after deletion) and skip the step.

Just ran into this problem as well, when re-orchestrating a host the changes don't quite manage to propagate from the Infoblox master to the SOA NSes quick enough for foreman.
Which meant dropped PTR records, even losing several A and AAAA records for a few machines.

Is there any particular reason why Foreman doesn't use the naive workflow of "Delete, and then always create anew"?

In foreman/app/models/concerns/dns_interface.rb on line 37;

  def recreate_dns_record(type)
    return true if dns_record(type).nil? || dns_record(type).valid?
    set_dns_record(type)
  end

Wouldn't it be possible to just skip the second check? Since the function is only ever called right after - successfully - deleting the record in question.

Actions #5

Updated by Dan Ports over 1 year ago

Sorry for the activity on such an old bug, but I ran into this same situation today on 3.5.0-rc1 and it was thoroughly frustrating. In my case, the cause was (I think; haven't verified) replication lag between two FreeIPA DNS servers, causing a DNS record to appear present immediately after deleting it. The result was that each attempt at rebuilding caused a different, seemingly random set of records, both PTR and A, to be lost.

It doesn't help that there's nothing informative in either the Foreman or proxy logs - just the old records being deleted and no indication that re-creating them is being skipped or why.

I want to re-emphasize this suggestion (from 5 years ago!):

Alexander Olofsson wrote:

Wouldn't it be possible to just skip the second check? Since the function is only ever called right after - successfully - deleting the record in question.

I tried making this change locally, and it resolved the issue.

Actions #6

Updated by Ewoud Kohl van Wijngaarden about 1 year ago

I always wondered about conflicts in Foreman. I share similar opinions as Alexander.

IMHO Foreman itself shouldn't do any DNS conflict detection. If anything, it should be delegated to the correct Smart Proxy which can have an accurate view. It can bypass DNS caches by going straight to the source.

Additionally the Smart Proxy should implement PUT /dns/:value/:type and Foreman should use that instead of DELETE + POST to ensures a certain state. Many DNS servers have an equivalent way, which can make updates more efficient. A single transaction also reduces the number of DNS zone updates, which helps with replication. #35087 is about that.

We should consider rollbacks, but a PUT method could make that more reliable as well.

Actions #7

Updated by Ewoud Kohl van Wijngaarden about 1 year ago

  • Related to Support #35087: DNS POST: how to force record creation added
Actions

Also available in: Atom PDF