Bug #687
closed
rake reports:expire abuses memory and network bandwidth
Description
Noticed this specifically from the reports:expire process:
SELECT * FROM `reports` WHERE (created_at < '2011-02-22 15:52:29' and status = 0)
There is no need at all to send the entire row. With thousands of hosts with 50 reports a day, this job sends GB of data over the network from the sql server (and into the rake processes memory) every time it is run just to delete reports. This should be optimized as it can cause some serious memory and network usage problems if processing a large number of reports.
Updated by Kal McFate about 13 years ago
Just for example:
I ran:
rake reports:expire days=12
followed by:
rake reports:expire days=10
results in:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5830 root 19 0 1300m 1.2g 3484 R 95.7 30.7 28:56.57 /usr/bin/ruby /usr/bin/rake reports:expire days=10
on 2 days of data.
30 minutes, and still going.
Updated by Ohad Levy about 13 years ago
yes you are correct.
the main reason we need to pull all reports is to know if we need to delete the related log, message and source records.
maybe we could improve the finder sql to make it faster.
one thing we could do quickly, is simply to read 1000 records or so each time instead of loading them all into memory.
Updated by Ohad Levy over 12 years ago
- Category set to Database
- Status changed from New to Assigned
- Assignee set to Ohad Levy
- Target version set to 0.4
Updated by Ohad Levy over 12 years ago
- Status changed from Assigned to Closed
- % Done changed from 0 to 100
Applied in changeset 0de3b547b52b2e2046d1953ffb286afcf2e46413.