Actions
Bug #23038
closed
after I sync some Red Hat content, Content -> Sync Status is returning errors: We're sorry, but something went wrong.
Description
It is Nightly installation:
foreman-1.18.0-0.develop.201803222233gite73b804.el7.noarch
katello-3.7.0-1.nightly.el7.noarch
Using hammer I have uploaded manifest and enabled some repos:
hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 7 Server (Kickstart)" --releasever "7.4" --basearch "x86_64" hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 7 Server (RPMs)" --releasever "7.4" --basearch "x86_64" hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 7 Server (RPMs)" --releasever "7Server" --basearch "x86_64" hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Satellite Tools 6.2 (for RHEL 6 Server) (RPMs)" --basearch "x86_64" hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Satellite Tools 6.3 (for RHEL 7 Server) (RPMs)" --basearch "x86_64" hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 7 Server - RH Common (RPMs)" --basearch "x86_64" --releasever 7Server hammer -u <user> -p <pass> repository-set enable --organization "Default Organization" --product "Red Hat Enterprise Linux Server" --name "Red Hat Enterprise Linux 7 Server - RH Common (RPMs)" --basearch "x86_64" --releasever 7.4
and then in webUI on Content -> Sync Status I have checked all of them except "Sat Tools 6.2" and clicked "Synchronize". I have kept it running over night and then in the morning I have refreshed the page and got error. There was this in the logs:
2018-03-28T02:25:43 [I|app|] Started GET "/katello/sync_management" for 10.40.204.104 at 2018-03-28 02:25:43 -0400
2018-03-28T02:25:43 [I|app|e3a03] Processing by Katello::SyncManagementController#index as HTML
2018-03-28T02:25:43 [I|app|e3a03] Current user: admin (administrator)
2018-03-28T02:25:44 [I|app|] Started GET "/notification_recipients" for 10.40.204.104 at 2018-03-28 02:25:44 -0400
2018-03-28T02:25:44 [I|app|3a337] Processing by NotificationRecipientsController#index as JSON
2018-03-28T02:25:44 [I|app|3a337] Current user: admin (administrator)
2018-03-28T02:25:44 [I|app|3a337] Completed 200 OK in 17ms (Views: 0.4ms | ActiveRecord: 2.2ms)
2018-03-28T02:25:54 [I|app|] Started GET "/notification_recipients" for 10.40.204.104 at 2018-03-28 02:25:54 -0400
2018-03-28T02:25:54 [I|app|5f6bc] Processing by NotificationRecipientsController#index as JSON
2018-03-28T02:25:54 [I|app|5f6bc] Current user: admin (administrator)
2018-03-28T02:25:54 [I|app|5f6bc] Completed 200 OK in 14ms (Views: 0.3ms | ActiveRecord: 1.7ms)
2018-03-28T02:26:05 [I|app|] Started GET "/notification_recipients" for 10.40.204.104 at 2018-03-28 02:26:05 -0400
2018-03-28T02:26:05 [I|app|0e649] Processing by NotificationRecipientsController#index as JSON
2018-03-28T02:26:05 [I|app|0e649] Current user: admin (administrator)
2018-03-28T02:26:05 [I|app|0e649] Completed 200 OK in 15ms (Views: 0.2ms | ActiveRecord: 1.7ms)
2018-03-28T02:26:13 [E|kat|e3a03] RestClient.post "https://foreman-nightly.example.com/pulp/api/v2/repositories/search/", "{\"criteria\":{\"filters\":{\"id\":{\"$in\":[\"2ecd0ed2-f026-4d1b-9ca0-c790078c7570\",\"f446135d-0055-4c94-b8d9-6550e8a5adf2\",\"29e3839d-e084-488e-89db-045597028551\",\"98c9f87f-4d02-4eba-89e1-c8114e8c6753\",\"af54dd5e-91ce-4776-beff-74b9e9d84ba8\",\"7cdf2dca-bb73-4b04-877a-77e677b88b57\",\"5049d7fc-c314-47ef-8720-52b22b45bc97\"]}}}}", "Accept"=>"*/*", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"314", "User-Agent"=>"rest-client/2.0.1 (linux-gnu x86_64) ruby/2.4.0p0", "accept"=>"application/json", "content_type"=>"application/json"
| \n# => 500 InternalServerError | application/json 2904 bytes
|
2018-03-28T02:26:13 [W|app|e3a03] Action failed
| RestClient::InternalServerError: 500 Internal Server Error
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:223:in `exception_with_response'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/abstract_response.rb:103:in `return!'
| /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:81:in `block in get_response'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:807:in `process_result'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:725:in `block in transmit'
| /opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in `start'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute'
| /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:67:in `post'
| /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response'
| /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:66:in `search'
| /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/extensions/repository.rb:96:in `search_by_repository_ids'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/models/katello/glue/pulp/repos.rb:53:in `prepopulate!'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/helpers/katello/sync_management_helper.rb:38:in `collect_repos'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/app/controllers/katello/sync_management_controller.rb:29:in `index'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/basic_implicit_render.rb:4:in `send_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:186:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rendering.rb:30:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:20:in `block in process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:108:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/application_shared.rb:15:in `set_timezone'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.6.0/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.6.0/lib/audited/sweeper.rb:14:in `around'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:135:in `run_callbacks'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/callbacks.rb:19:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/rescue.rb:20:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:32:in `block in process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `block in instrument'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications/instrumenter.rb:21:in `instrument'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:166:in `instrument'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/instrumentation.rb:30:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal/params_wrapper.rb:252:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.4/lib/active_record/railties/controller_runtime.rb:22:in `process_action'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/abstract_controller/base.rb:124:in `process'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionview-5.1.4/lib/action_view/rendering.rb:30:in `process'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:189:in `dispatch'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_controller/metal.rb:253:in `dispatch'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:49:in `dispatch'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:31:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/mapper.rb:17:in `block in <class:Constraints>'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/mapper.rb:46:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:50:in `block in serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `each'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/journey/router.rb:33:in `serve'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/routing/route_set.rb:834:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/static_dispatcher.rb:65:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/extractor/recorder.rb:136:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /usr/share/foreman/lib/middleware/telemetry.rb:9:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.6/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
| /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:8:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/conditional_get.rb:25:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/head.rb:12:in `call'
| /usr/share/foreman/lib/middleware/logging_context.rb:23:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:232:in `context'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/session/abstract/id.rb:226:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/cookies.rb:613:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:26:in `block in call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/callbacks.rb:97:in `run_callbacks'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/callbacks.rb:24:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/debug_exceptions.rb:59:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/show_exceptions.rb:31:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:36:in `call_app'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/rack/logger.rb:26:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/remote_ip.rb:79:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/request_id.rb:25:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0/lib/katello/prevent_json_parsing.rb:12:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/method_override.rb:22:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/runtime.rb:22:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache/strategy/local_cache_middleware.rb:27:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/executor.rb:12:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/actionpack-5.1.4/lib/action_dispatch/middleware/static.rb:125:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/sendfile.rb:111:in `call'
| /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-3.4.1/lib/secure_headers/middleware.rb:12:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/engine.rb:522:in `call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `public_send'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/railties-5.1.4/lib/rails/railtie.rb:185:in `method_missing'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:68:in `block in call'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `each'
| /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/rack-2.0.3/lib/rack/urlmap.rb:53:in `call'
| /usr/share/passenger/phusion_passenger/rack/thread_handler_extension.rb:74:in `process_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:141:in `accept_and_process_next_request'
| /usr/share/passenger/phusion_passenger/request_handler/thread_handler.rb:109:in `main_loop'
| /usr/share/passenger/phusion_passenger/request_handler.rb:455:in `block (3 levels) in start_threads'
| /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2018-03-28T02:26:13 [I|app|e3a03] Rendering common/500.html.erb within katello/layouts/katello
2018-03-28T02:26:13 [I|app|e3a03] Rendered common/500.html.erb within katello/layouts/katello (2.7ms)
2018-03-28T02:26:13 [I|app|e3a03] Completed 500 Internal Server Error in 30467ms (ActiveRecord: 22.8ms)
2018-03-28T02:26:13 [F|app|]
2018-03-28T02:26:13 [F|app|] ActionView::Template::Error (wrong number of arguments (given 1, expected 0)):
2018-03-28T02:26:13 [F|app|] 1: <%= alert :header => _("Oops, we're sorry but something went wrong"), :class => 'alert-danger',
| 2: :actions => link_to(_('Back'), main_app.root_path, :class => 'btn btn-default'),
| 3: :text => h(exception.message),
| 4: :close => false %>
2018-03-28T02:26:13 [F|app|]
2018-03-28T02:26:13 [F|app|] app/views/common/500.html.erb:1:in `_9a56ec1afd58d796dd52bee6713eda52'
| app/controllers/application_controller.rb:323:in `generic_exception'
| lib/middleware/telemetry.rb:9:in `call'
| lib/middleware/catch_json_parse_errors.rb:8:in `call'
| lib/middleware/logging_context.rb:23:in `call'
| katello (3.7.0) lib/katello/prevent_json_parsing.rb:12:in `call'
Updated by Jan Hutař about 6 years ago
Looks like these two services were down:
# katello-service status [...] Some services failed to status: mongod,pulp_celerybeat # systemctl status mongod pulp_celerybeat ● mongod.service - High-performance, schema-free document-oriented database Loaded: loaded (/usr/lib/systemd/system/mongod.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since Tue 2018-03-27 18:11:02 EDT; 8h ago Main PID: 15435 (code=exited, status=14) Mar 27 10:01:14 foreman-nightly.example.com systemd[1]: Starting High-performance, schema-free document-oriented database... Mar 27 10:01:14 foreman-nightly.example.com mongod[15433]: about to fork child process, waiting until server is ready for connections. Mar 27 10:01:14 foreman-nightly.example.com mongod[15433]: forked process: 15435 Mar 27 10:01:15 foreman-nightly.example.com mongod[15433]: child process started successfully, parent exiting Mar 27 10:01:15 foreman-nightly.example.com systemd[1]: Started High-performance, schema-free document-oriented database. Mar 27 18:11:02 foreman-nightly.example.com systemd[1]: mongod.service: main process exited, code=exited, status=14/n/a Mar 27 18:11:02 foreman-nightly.example.com systemd[1]: Unit mongod.service entered failed state. Mar 27 18:11:02 foreman-nightly.example.com systemd[1]: mongod.service failed. ● pulp_celerybeat.service - Pulp's Celerybeat Loaded: loaded (/usr/lib/systemd/system/pulp_celerybeat.service; enabled; vendor preset: disabled) Active: inactive (dead) since Tue 2018-03-27 18:12:06 EDT; 8h ago Main PID: 18354 (code=exited, status=0/SUCCESS) Mar 27 18:12:06 foreman-nightly.example.com pulp[18354]: celery.beat:CRITICAL: (18354-79872) self._error_message(selector)) Mar 27 18:12:06 foreman-nightly.example.com pulp[18354]: celery.beat:CRITICAL: (18354-79872) ServerSelectionTimeoutError: localhost:27017: [Errno 111] Connection refused Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: celery beat v3.1.17 (Cipater) is starting. Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: __ - ... __ - _ Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: Configuration -> Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: . broker -> qpid://localhost:5671// Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: . loader -> celery.loaders.app.AppLoader Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: . scheduler -> pulp.server.async.scheduler.Scheduler Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: . logfile -> [stderr]@%INFO Mar 27 18:12:06 foreman-nightly.example.com celery[18354]: . maxinterval -> now (0s)
So looks like these services failed sometime short after sync start.
Updated by Jan Hutař about 6 years ago
- Status changed from New to Rejected
Err, this was about a disk space :-/
Investigation showed that mongo is not starting /var/log/mongodb/mongodb.log:
2018-03-28T03:19:21.455-0400 ***** SERVER RESTARTED *****
2018-03-28T03:19:21.466-0400 [initandlisten] MongoDB starting : pid=20775 port=27017 dbpath=/var/lib/mongodb 64-bit host=foreman-nightly.example.com
2018-03-28T03:19:21.467-0400 [initandlisten] db version v2.6.12
2018-03-28T03:19:21.467-0400 [initandlisten] git version: nogitversion
2018-03-28T03:19:21.467-0400 [initandlisten] OpenSSL version: OpenSSL 1.0.2k-fips 26 Jan 2017
2018-03-28T03:19:21.467-0400 [initandlisten] build info: Linux buildvm-23.phx2.fedoraproject.org 4.14.11-300.fc27.x86_64 #1 SMP Wed Jan 3 13:52:28 UTC 2018 x86_64 BOOST_LIB_VERSION=1_53
2018-03-28T03:19:21.467-0400 [initandlisten] allocator: tcmalloc
2018-03-28T03:19:21.467-0400 [initandlisten] options: { command: [ "run" ], config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1" }, processManagement: { fork: true, pidFilePath: "/var/run/mongodb/mongod.pid" }, security: { authorization: "disabled" }, storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongodb.log", quiet: true } }
2018-03-28T03:19:21.468-0400 [initandlisten]
2018-03-28T03:19:21.468-0400 [initandlisten] ** WARNING: Readahead for /var/lib/mongodb is set to 4096KB
2018-03-28T03:19:21.468-0400 [initandlisten] ** We suggest setting it to 256KB (512 sectors) or less
2018-03-28T03:19:21.468-0400 [initandlisten] ** http://dochub.mongodb.org/core/readahead
2018-03-28T03:19:21.472-0400 [initandlisten] journal dir=/var/lib/mongodb/journal
2018-03-28T03:19:21.472-0400 [initandlisten] recover : no journal files present, no recovery needed
2018-03-28T03:19:21.472-0400 [initandlisten]
2018-03-28T03:19:21.472-0400 [initandlisten] ERROR: Insufficient free space for journal files
2018-03-28T03:19:21.472-0400 [initandlisten] Please make at least 3379MB available in /var/lib/mongodb/journal or use --smallfiles
2018-03-28T03:19:21.472-0400 [initandlisten]
2018-03-28T03:19:21.472-0400 [initandlisten] exception in initAndListen: 15926 Insufficient free space for journals, terminating
2018-03-28T03:19:21.473-0400 [initandlisten] dbexit:
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: going to close listening sockets...
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: going to flush diaglog...
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: going to close sockets...
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: waiting for fs preallocator...
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: lock for final commit...
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: final commit...
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: closing all files...
2018-03-28T03:19:21.473-0400 [initandlisten] closeAllFiles() finished
2018-03-28T03:19:21.473-0400 [initandlisten] journalCleanup...
2018-03-28T03:19:21.473-0400 [initandlisten] removeJournalFiles
2018-03-28T03:19:21.473-0400 [initandlisten] shutdown: removing fs lock...
2018-03-28T03:19:21.473-0400 [initandlisten] dbexit: really exiting now
Updated by John Mitsch about 6 years ago
- translation missing: en.field_release set to 166
Actions