Support #11349: Sychronize capsule content never finishes - Katello - Foreman

Actions

Copy link

Support #11349

closed

Sychronize capsule content never finishes

Added by Sameer Syed over 8 years ago. Updated almost 6 years ago.

Status:

Resolved

Priority:

High

Assignee:

Category:

Target version:

Katello Recycle Bin

Triaged:

Fixed in Releases:

Found in Releases:

Description

Synchronization from katello to external capsule takes hours and never finishes. From the foreman task it shows as stuck at 50%. I've restarted goferd on the capsule, that did not help. I started the sync manually:

hammer capsule content synchronize --id <ID>

I also see the following errors in the katello host logs (sanitized), may not be related.

Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:525 - trying: katello.example.com:5671
Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:443 - recoverable error[attempt 105]: [Errno 113] No route to host
Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:445 - sleeping 120 seconds

Here is netstat out from katello host:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:28017 0.0.0.0:* LISTEN 32919/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3331/sshd
tcp 0 0 127.0.0.1:48632 0.0.0.0:* LISTEN 31398/Passenger Rac
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3376/sendmail
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 16876/httpd
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 25378/ruby
tcp 0 0 0.0.0.0:836 0.0.0.0:* LISTEN 3199/ypbind
tcp 0 0 0.0.0.0:5671 0.0.0.0:* LISTEN 33114/qpidd
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 3313/snmpd
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 33114/qpidd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 32919/mongod
tcp 0 0 0.0.0.0:8140 0.0.0.0:* LISTEN 16876/httpd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3074/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 16876/httpd
udp 0 0 0.0.0.0:701 0.0.0.0:* 3074/rpcbind
udp 0 0 0.0.0.0:833 0.0.0.0:* 3199/ypbind
udp 0 0 0.0.0.0:33492 0.0.0.0:* 3024/rsyslogd
udp 0 0 0.0.0.0:111 0.0.0.0:* 3074/rpcbind
udp 0 0 10.224.165.45:123 0.0.0.0:* 3352/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 3352/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 3352/ntpd
udp 0 0 0.0.0.0:797 0.0.0.0:* 3199/ypbind
udp 0 0 0.0.0.0:161 0.0.0.0:* 3313/snmpd
udp 0 0 0.0.0.0:39599 0.0.0.0:* 3024/rsyslogd

netstat from capsule host:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:28017 0.0.0.0:* LISTEN 35652/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 7381/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7426/sendmail
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 29370/httpd
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 26142/ruby
tcp 0 0 0.0.0.0:645 0.0.0.0:* LISTEN 7248/ypbind
tcp 0 0 0.0.0.0:5671 0.0.0.0:* LISTEN 35854/qpidd
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 7363/snmpd
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 35854/qpidd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 35652/mongod
tcp 0 0 127.0.0.1:35052 0.0.0.0:* LISTEN 29681/Passenger Rac
tcp 0 0 0.0.0.0:8140 0.0.0.0:* LISTEN 29370/httpd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3054/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 29370/httpd
udp 0 0 0.0.0.0:742 0.0.0.0:* 7248/ypbind
udp 0 0 0.0.0.0:111 0.0.0.0:* 3054/rpcbind
udp 0 0 0.0.0.0:47219 0.0.0.0:* 3021/rsyslogd
udp 0 0 10.238.24.20:123 0.0.0.0:* 7402/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 7402/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 7402/ntpd
udp 0 0 0.0.0.0:642 0.0.0.0:* 7248/ypbind
udp 0 0 0.0.0.0:161 0.0.0.0:* 7363/snmpd
udp 0 0 0.0.0.0:48549 0.0.0.0:* 3021/rsyslogd
udp 0 0 0.0.0.0:681 0.0.0.0:* 3054/rpcbind

Please advise.

Actions

Copy link

Updated by Sameer Syed over 8 years ago

Version of katello-agent:

katello-agent-2.1.4-1.el6.noarch

Sameer Syed wrote:

Synchronization from katello to external capsule takes hours and never finishes. From the foreman task it shows as stuck at 50%. I've restarted goferd on the capsule, that did not help. I started the sync manually:

hammer capsule content synchronize --id <ID>

I also see the following errors in the katello host logs (sanitized), may not be related.

Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:525 - trying: katello.example.com:5671
Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:443 - recoverable error[attempt 105]: [Errno 113] No route to host
Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:445 - sleeping 120 seconds

Here is netstat out from katello host:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:28017 0.0.0.0:* LISTEN 32919/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 3331/sshd
tcp 0 0 127.0.0.1:48632 0.0.0.0:* LISTEN 31398/Passenger Rac
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 3376/sendmail
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 16876/httpd
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 25378/ruby
tcp 0 0 0.0.0.0:836 0.0.0.0:* LISTEN 3199/ypbind
tcp 0 0 0.0.0.0:5671 0.0.0.0:* LISTEN 33114/qpidd
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 3313/snmpd
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 33114/qpidd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 32919/mongod
tcp 0 0 0.0.0.0:8140 0.0.0.0:* LISTEN 16876/httpd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3074/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 16876/httpd
udp 0 0 0.0.0.0:701 0.0.0.0:* 3074/rpcbind
udp 0 0 0.0.0.0:833 0.0.0.0:* 3199/ypbind
udp 0 0 0.0.0.0:33492 0.0.0.0:* 3024/rsyslogd
udp 0 0 0.0.0.0:111 0.0.0.0:* 3074/rpcbind
udp 0 0 10.224.165.45:123 0.0.0.0:* 3352/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 3352/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 3352/ntpd
udp 0 0 0.0.0.0:797 0.0.0.0:* 3199/ypbind
udp 0 0 0.0.0.0:161 0.0.0.0:* 3313/snmpd
udp 0 0 0.0.0.0:39599 0.0.0.0:* 3024/rsyslogd

netstat from capsule host:

netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:28017 0.0.0.0:* LISTEN 35652/mongod
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 7381/sshd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 7426/sendmail
tcp 0 0 0.0.0.0:443 0.0.0.0:* LISTEN 29370/httpd
tcp 0 0 0.0.0.0:9090 0.0.0.0:* LISTEN 26142/ruby
tcp 0 0 0.0.0.0:645 0.0.0.0:* LISTEN 7248/ypbind
tcp 0 0 0.0.0.0:5671 0.0.0.0:* LISTEN 35854/qpidd
tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 7363/snmpd
tcp 0 0 0.0.0.0:5672 0.0.0.0:* LISTEN 35854/qpidd
tcp 0 0 127.0.0.1:27017 0.0.0.0:* LISTEN 35652/mongod
tcp 0 0 127.0.0.1:35052 0.0.0.0:* LISTEN 29681/Passenger Rac
tcp 0 0 0.0.0.0:8140 0.0.0.0:* LISTEN 29370/httpd
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 3054/rpcbind
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 29370/httpd
udp 0 0 0.0.0.0:742 0.0.0.0:* 7248/ypbind
udp 0 0 0.0.0.0:111 0.0.0.0:* 3054/rpcbind
udp 0 0 0.0.0.0:47219 0.0.0.0:* 3021/rsyslogd
udp 0 0 10.238.24.20:123 0.0.0.0:* 7402/ntpd
udp 0 0 127.0.0.1:123 0.0.0.0:* 7402/ntpd
udp 0 0 0.0.0.0:123 0.0.0.0:* 7402/ntpd
udp 0 0 0.0.0.0:642 0.0.0.0:* 7248/ypbind
udp 0 0 0.0.0.0:161 0.0.0.0:* 7363/snmpd
udp 0 0 0.0.0.0:48549 0.0.0.0:* 3021/rsyslogd
udp 0 0 0.0.0.0:681 0.0.0.0:* 3054/rpcbind

Please advise.

Actions

Copy link

Updated by Dominic Cleal over 8 years ago

Tracker changed from Support to Bug
Project changed from Foreman to Katello
Triaged set to No

Actions

Copy link

Updated by Eric Helms over 8 years ago

translation missing: en.field_release set to 77
Triaged changed from No to Yes

Actions

Copy link

Updated by Stephen Benjamin over 8 years ago

translation missing: en.field_release deleted (77)

This indicates you have a connectivity problem:

Aug 13 20:03:08 katello.example.com goferd: [WARNING][Thread-2] qpid.messaging:443 - recoverable error[attempt 105]: [Errno 113] No route to host

Such as a firewall or something preventing the agent on the capsule to talk to port 5671 on the Katello. That might help you get things going again, but it doesn't look like a bug.

It's also Katello 2.1, I'd recommend upgrading to 2.3 when you can.

Actions

Copy link