Project

General

Profile

Actions

Bug #5568

closed

Possible memory leak in trends

Added by Lukas Zapletal almost 10 years ago. Updated over 5 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Trends
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

By Sean Alderman (https://groups.google.com/forum/#!msg/foreman-users/hi7TagIRE8A/5ZRzlI8ChxEJ):

I am collecting a few trends in foreman using host facts. I'm using a cron entry for the foreman user: /usr/sbin/foreman-rake trends:counter every half hour.

My foreman machine is running puppet, puppetdb, foreman, foreman-proxy, and postgres, 2 vCPU and 6GB of RAM. It seems that the rake commands are running away with all the memory and swap. Is there a way to control this?

Here's what top shows currently:

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                               
30002 foreman   20   0 6602m 3.3g  104 D 13.3 58.3  17:06.22 /opt/rh/ruby193/root/usr/bin/ruby /opt/rh/ruby193/root/usr/bin/rake trends:clean                                                                      

Seems like a lot of RAM and accumulated CPU time. I can add more RAM, but other than rake commands like this one, the server doesn't seem to utilize the memory allocated.


Related issues 1 (0 open1 closed)

Related to Foreman - Bug #4114: Trends don't scale well and become very slowDuplicate01/17/2014Actions
Actions #1

Updated by Dominic Cleal almost 10 years ago

  • Related to Bug #4114: Trends don't scale well and become very slow added
Actions #2

Updated by Edson Manners over 9 years ago

Seem to have a similar issue. Currently running 1.4.5 on RHEL 6.5.
We have 545 hosts and I'm not running and anytrends manually foreman seems ot be running these trends on it's own.
I'm running this in a KVM VM with 6 cores and 16GB RAM ot this point but it's obviously never enough.
I'll be upgrading through 1.5 and 1.6 in the coming days and looking to see if that helps.

Top show this after 'service foreman stop; service httpd stop'
30143 foreman 20 0 8407m 7.9g 6544 D 11.6 50.9 42:44.19 rake

I have to 'kill -9' the rake process as it doesn't die on it's own.

'ps -lef | grep 30143'
/opt/rh/ruby193/root/usr/bin/ruby /opt/rh/ruby193/root/usr/bin/rake trends:clean

Actions #3

Updated by Shimon Shtein about 9 years ago

Can it be related to the amount of data in the trend_counters table?

Actions #4

Updated by Jon McKenzie almost 9 years ago

We're running into this issue too. I haven't done a ton with ActiveRecord, but the issue are these lines in the trends:clean rake task:

counts = TrendCounter.group([:trend_id, :created_at]).count

dupes = counts.select{ |attrs, count| count > 1}

Rather than doing the aggregation inside the database, the task first executes a very expensive GROUP and then filters the results it needs in plain Ruby.

Again, not an ActiveRecord expert, but I think replacing those two lines with the following should significantly speed it up:

dupes = TrendCounter.having('count(*) > 1').group([:trend_id, :created_at]).count

This translates the SQL from this:

SELECT COUNT(*) AS count_all, trend_id AS trend_id, created_at AS created_at FROM "trend_counters" GROUP BY trend_id, created_at ORDER BY created_at

...to this:

SELECT COUNT(*) AS count_all, trend_id AS trend_id, created_at AS created_at FROM "trend_counters" GROUP BY trend_id, created_at HAVING count(*) > 1 ORDER BY created_at
Actions #5

Updated by Jon McKenzie almost 9 years ago

Huh, I must have misread this issue initially, which is for trends:counter not trends:clean. Oh well, I submitted a PR for the trends:clean issue anyways

Actions #6

Updated by Dominic Cleal almost 9 years ago

  • Status changed from Assigned to Ready For Testing
  • Assignee deleted (Lukas Zapletal)
  • Pull request https://github.com/theforeman/foreman/pull/2365 added
  • Pull request deleted ()
Actions #7

Updated by Jon McKenzie almost 9 years ago

  • Status changed from Ready For Testing to Closed
  • % Done changed from 0 to 100
Actions #8

Updated by Dominic Cleal almost 9 years ago

  • Assignee set to Jon McKenzie
  • translation missing: en.field_release set to 50
Actions

Also available in: Atom PDF