Project

General

Profile

Actions

Bug #21599

closed

duplicate key value violates unique constraint "index_fact_names_on_name_and_type" on simulatenous host register

Added by Shimon Shtein over 6 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Category:
Facts
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

Cloned from https://bugzilla.redhat.com/show_bug.cgi?id=1474293

Description of problem:
Our automation hit yet another race condition on registering the hosts simultaneously (2 hosts).
See attached foreman-debug:
production.log - see requests: 4c6a15a5 and 64762ef4 (which failed) (2017-07-20 21:05:54)

2017-07-20 21:05:51 64762ef4 [app] [I] Started POST "/rhsm/consumers?owner=Ag72NG&activation_keys=IsRwhiODa7" for 10.8.30.229 at 2017-07-20 21:05:51 -0400
2017-07-20 21:05:51 64762ef4 [app] [I] Processing by Katello::Api::Rhsm::CandlepinProxiesController#consumer_activate as JSON
...
...
2017-07-20 21:05:54 [audit] [I] [140318963445328.lab.eng.rdu2.redhat.com] added 153 (3963.0ms)
2017-07-20 21:05:54 [app] [I] Import facts for '140318963445328.lab.eng.rdu2.redhat.com' completed. Added: 153, Updated: 0, Deleted 0 facts
2017-07-20 21:05:54 [app] [E] Fact dmi::system::uuid could not be imported because of PG::Error: ERROR: duplicate key value violates unique constraint "index_fact_names_on_name_and_type" | : INSERT INTO "fact_names" ("compose", "created_at", "name", "short_name", "type", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"
2017-07-20 21:05:54 [app] [E] Fact distribution::name could not be imported because of PG::Error: ERROR: current transaction is aborted, commands ignored until end of transaction block | : SELECT "fact_names".* FROM "fact_names" WHERE "fact_names"."type" IN ('Katello::RhsmFactName') AND "fact_names"."name" = 'distribution' AND "fact_names"."type" = 'Katello::RhsmFactName' ORDER BY fact_names.name LIMIT 1
...
...
2017-07-20 21:05:55 [audit] [I] [140599713026320.lab.eng.rdu2.redhat.com] added 166 (1966.6ms)
2017-07-20 21:05:55 [app] [W] Error during fact import for 140599713026320.lab.eng.rdu2.redhat.com | ActiveRecord::StatementInvalid: PG::Error: ERROR: current transaction is aborted, commands ignored until end of transaction block | : SELECT "fact_names".* FROM "fact_names" WHERE "fact_names"."type" IN ('Katello::RhsmFactName') AND "fact_names"."name" = 'memor | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `ex | /opt/rh/rh-ror41/root/usr/share/gems/gems/activerecord-4.1.5/lib/active_record/connection_adapters/postgresql_adapter.rb:822:in `bl
...
...
2017-07-20 21:05:55 [foreman-tasks/action] [E] ERF50-2486 [Foreman::WrappedException]: Import of facts failed for host 14059971302632 | /usr/share/foreman/app/services/fact_importer.rb:48:in `import!'
... | /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-1.0.0/lib/concurrent/executor/ruby_thread_pool_executor.rb:304:in `blo | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `call' | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-1.8.2/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_c
2017-07-20 21:05:56 64762ef4 [app] [I] Completed 500 Internal Server Error in 5045ms (Views: 0.6ms | ActiveRecord: 92.5ms)
2017-07-20 21:05:57 4c6a15a5 [app] [I] Completed 200 OK in 9308ms (Views: 2.9ms | ActiveRecord: 45.2ms)

Version-Release number of selected component (if applicable):
6.2.11-1

How reproducible:
sometimes

Steps to Reproduce:
hard to reproduce, however:
1. get 2 different hosts register to the same satellite at the same time, with the same org, env and activation key
2. making katello more busy might help, in our case, there were some POST requests to Environments and Content Views made just after the register requests.

Actual results:
duplicate key value violates unique constraint "index_fact_names_on_name_and_type"


Related issues 2 (0 open2 closed)

Related to Foreman - Bug #9016: Fact import code consumes lot of memoryClosedLukas Zapletal01/18/2015Actions
Related to Foreman - Bug #21776: Fix fact names transaction handlingClosedShimon Shtein11/27/2017Actions
Actions #1

Updated by The Foreman Bot over 6 years ago

  • Status changed from New to Ready For Testing
  • Assignee set to Shimon Shtein
  • Pull request https://github.com/theforeman/foreman/pull/4996 added
Actions #2

Updated by Marek Hulán over 6 years ago

  • Related to Bug #9016: Fact import code consumes lot of memory added
Actions #3

Updated by The Foreman Bot over 6 years ago

  • Pull request https://github.com/theforeman/foreman/pull/5007 added
Actions #4

Updated by Ivan Necas over 6 years ago

  • Category set to Facts
Actions #5

Updated by Lukas Zapletal over 6 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #6

Updated by Lukas Zapletal over 6 years ago

  • Status changed from Closed to Assigned

THERE ARE TWO PATCHES ASSOCIATED.

One is quick and dirty patch of removing explicit SQL transactions, this should be easily backportable in case you are hit by this bug:
https://github.com/theforeman/foreman/pull/5007

The other one is much more complex, it keeps transactions and only pulls fact name creation out:
https://github.com/theforeman/foreman/pull/5007

Actions #7

Updated by Marek Hulán over 6 years ago

  • Status changed from Assigned to Closed
  • Assignee changed from Shimon Shtein to Lukas Zapletal
  • translation missing: en.field_release set to 296
  • Pull request deleted (https://github.com/theforeman/foreman/pull/5007)

Marking as 1.17.0 as the transaction was dropped, the second PR is more complicated and it's not clear whether it makes it into 1.17 so I'm removing it for now. If it will make it in before branching, it can be readded, otherwise a new redmine issue should be created.

Actions #8

Updated by Shimon Shtein over 6 years ago

  • Pull request https://github.com/theforeman/foreman/pull/5007 added
  • Pull request deleted (https://github.com/theforeman/foreman/pull/4996)

Marek, I think you meant to remove https://github.com/theforeman/foreman/pull/4996 and leave the transaction dropping (https://github.com/theforeman/foreman/pull/5007)

Actions #9

Updated by Lukas Zapletal over 6 years ago

  • Related to Bug #21776: Fix fact names transaction handling added
Actions

Also available in: Atom PDF