Bug #22671
closed
Sync plans not running automatically
Added by Philipp Mueller about 6 years ago. Updated about 5 years ago.
Description
Hi,
on our Katello 3.5.1 system, sync plans are not running as scheduled. When i click "sync now" its working.
Deleting or adding new sync plans also doenst work.
Kind regards,
Philipp
Updated by Philipp Mueller about 6 years ago
[root@XXX ~]# su - postgres c "psql -d foreman -c 'select * from katello_sync_plans;'" --------------------------+---------------------+----------+-----------------+----------------------------+----------------------------+---------
id | name | description | sync_date | interval | organization_id | created_at | updated_at | enabled
---
5 | Daily | | 2017-11-08 11:21:00 | daily | 1 | 2017-11-08 08:22:38.246492 | 2017-12-29 15:34:42.042774 | t
2 | daily-gedis | | 2017-06-08 00:00:00 | daily | 29 | 2017-06-08 08:22:18.763727 | 2017-06-08 08:22:18.763727 | t
3 | daily | | 2017-06-22 02:35:00 | daily | 30 | 2017-06-22 13:17:01.277477 | 2017-06-22 13:17:01.277477 | t
6 | test | | 2017-11-15 11:11:00 | hourly | 1 | 2017-11-16 09:12:07.282427 | 2017-11-16 09:12:07.282427 | t
[root@xxx~]# pulp-admin u admin -p $pulpAdminPassword rpm repo sync schedules list --repo-id 1-Internet-CentOS7-v5_0-rs-REMI-REMI7
---------------------------------------------------------------------
Schedules--------------------------------------------------------------------
There are no schedules defined for this operation.
Updated by Andrew Kofink about 6 years ago
- Assignee set to Andrew Kofink
- Target version set to 160
- translation missing: en.field_release set to 339
Updated by Philipp Mueller about 6 years ago
I have found a workaround for this problem.
When i checked the repo schedules with pulp-admin, it appeared, that they are running at the correct times according to the sync-plan, which was created in katello.
The mechanism, that pulp uses, to report the sync_complete status to katello via the rest-api was not working, due to a ssl-certificate, that could not be verified.
We are using custom ssl-certificates and have imported the root-ca to the trusted root CAs.
In the pulp log, after every successfully scheduled sync, pulp fails with this error:
Mar 12 10:54:47 hostname pulp: requests.packages.urllib3.connectionpool:INFO: [2dec033d] Starting new HTTPS connection (1): hostname.domain.told
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) HTTP Notification Failed
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) Traceback (most recent call last):
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/pulp/server/event/http.py", line 57, in _send_post
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) headers={'Content-Type': 'application/json'}, timeout=15)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/api.py", line 108, in post
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) return request('post', url, data=data, json=json, *kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/api.py", line 50, in request
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) response = session.request(method=method, url=url, **kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 464, in request
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) resp = self.send(prep, **send_kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 576, in send
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) r = adapter.send(request, **kwargs)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 431, in send
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) raise SSLError(e, request=request)
Mar 12 10:54:47 hostname pulp: pulp.server.event.http:ERROR: [2dec033d] (2432-66336) SSLError: [SSL: *CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579
The function for the post is found in: /usr/lib/python2.7/site-packages/pulp/server/event/http.py. After adding the parameter verify=false, the post works again and the Sync-Plans in the Katello frontend show their respective status.
def _send_post(notifier_config, json_body):
response = post(url, data=json_body, auth=auth,
headers={'Content-Type': 'application/json'}, timeout=15,*verify=False)*
Updated by Andrew Kofink about 6 years ago
- translation missing: en.field_release changed from 339 to 351
Updated by Martin Angermeier almost 6 years ago
With Katello 3.6.0 this problem seems to occur again.
Kind regards,
Martin Angermeier
Updated by Andrew Kofink almost 6 years ago
Thanks for the info. I'm also able to reproduce this on nightly Katello.
Updated by Andrew Kofink over 5 years ago
- Subject changed from Katello 3.5.1 - sync plans not running automatically to Sync plans not running automatically
- Assignee deleted (
Andrew Kofink) - Priority changed from Normal to High
- Target version changed from Katello 3.5.3 to Katello 3.9.0
- Triaged set to No
- Found in Releases Katello 3.5.0, Katello 3.5.1, Katello 3.5.2, Katello 3.5.3, Katello 3.6.0, Katello 3.6.1, Katello 3.7.0 added
This doesn't seem to be resolved in 3.6 or 3.7. I'm unsetting triaged so we'll review it this week in triage for inclusion in 3.9.
Updated by Andrew Kofink over 5 years ago
- Is duplicate of Bug #24083: Add an upgrade rake task for sync plans added
Updated by Andrew Kofink over 5 years ago
- Status changed from New to Duplicate
- Target version changed from Katello 3.9.0 to Katello Recycle Bin
- Triaged changed from No to Yes
Updated by Ludwig Bayerlein about 5 years ago
- Found in Releases Katello 3.7.1, Katello 3.8.1 added
This is a Bug in python-certifi. On our side, there was installed python-certifi with pip. We verified this with
# pip list|grep certifi
certifi 2018.10.15
Then I uninstalled it with
# pip uninstall certifi
And did a reinstall of python-requests
# yum reinstall -y python-requests
After this few steps, everything works great. The user @jsherrill explained that as a bug on the pip installed version of certifi, because it uses the wrong ca path.
Thanks for your support!