Bug #23061
closed
remote execution tasks stuck in status planned
Added by Christian Meißner about 6 years ago.
Updated about 6 years ago.
Description
After upgrading foreman to 1.16.0 we had some problems with duplicated database objects in dynflow schema. So we deleted all schema objects. Now REX doesn't work anymore. All planned jobs stuck in status "planned"
Any hints how we can get REX to work?
Files
Attachted you find a passenger.log.
- Project changed from Foreman to Foreman Remote Execution
- Priority changed from High to Normal
What exactly did you do to remove the objects?
- Related to Bug #12993: PostgreSQL DuplicateColumn Error added
are there some news for that issue? Is there a way to workaround the problem. We have some hundreds of machines which we have to manage via REX.
Hello, sorry for taking so long. So far I wasn't able to reproduce the issue you're having.
To get on the same page, you're on debian and were on foreman 1.15.x. You upgraded to 1.16 and got the duplicate keys error. So you solved that by dropping dynflow tables and then REX didn't work anymore. Is that correct?
A couple of questions to help us troubleshoot this. Have you restarted any services afterwards? If you go to FOREMAN_SERVER_URL/foreman_tasks/dynflow/status, is there any world marked as executor? If you check in the db, did the dynflow tables get created again?
Adam Ruzicka wrote:
To get on the same page, you're on debian and were on foreman 1.15.x. You upgraded to 1.16 and got the duplicate keys error. So you solved that by dropping dynflow tables and then REX didn't work anymore. Is that correct?
Yes obviously. Yesterday We update Foreman to 1.16.1 and get also the "duplicate keys" error. Here we solved the issue by downgrading the pg gem.
A couple of questions to help us troubleshoot this. Have you restarted any services afterwards? If you go to FOREMAN_SERVER_URL/foreman_tasks/dynflow/status, is there any world marked as executor? If you check in the db, did the dynflow tables get created again?
Attached you find the output of dynflow status.
And yes the dynflow tables were created again.
public | dynflow_actions | table | foreman
public | dynflow_coordinator_records | table | foreman
public | dynflow_delayed_plans | table | foreman
public | dynflow_envelopes | table | foreman
public | dynflow_execution_plans | table | foreman
public | dynflow_schema_info | table | foreman
public | dynflow_steps | table | foreman
All the worlds are client ones and are most likely part of the rails processes. However most of things is handled by a standalone service called foreman-tasks (or dynflowd on newer versions). Could you restart it? Another world record should show up FOREMAN_SERVER_URL/foreman_tasks/dynflow/status and it should be marked as executor. If it does, things should start moving again
Adam Ruzicka wrote:
All the worlds are client ones and are most likely part of the rails processes. However most of things is handled by a standalone service called foreman-tasks (or dynflowd on newer versions). Could you restart it? Another world record should show up FOREMAN_SERVER_URL/foreman_tasks/dynflow/status and it should be marked as executor. If it does, things should start moving again
There is no service called foreman-tasks or dynflowd on my system. I can only restart apache2 to restart the rail application. But after that restart nothing changed. See attachment.
I'm sorry, I was a bit off with the name, the service is apparently called ruby-foreman-tasks on deb-based systems. The service is provided by ruby-foreman-tasks package, which should get pulled in by remote execution.
Adam Ruzicka wrote:
I'm sorry, I was a bit off with the name, the service is apparently called ruby-foreman-tasks on deb-based systems. The service is provided by ruby-foreman-tasks package, which should get pulled in by remote execution.
This does the trick. But my question is now. Why does a system reboot not fix the problem?
- Status changed from New to Duplicate
- Is duplicate of Bug #20050: ruby-sequel-pg DEB pollutes Foreman ruby gem environment added
Also available in: Atom
PDF
Updated by 
Updated by Anonymous 
Updated by 
Updated by