Project

General

Profile

Actions

Bug #1592

closed

report::expire errors with millions of records

Added by Jacob McCann about 12 years ago. Updated almost 6 years ago.

Status:
Closed
Priority:
Normal
Assignee:
Category:
Reporting
Target version:
Difficulty:
Triaged:
Fixed in Releases:
Found in Releases:

Description

When running report::expire against a DB with millions of records in Reports the 'function' will fail (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/kH0ISyjoMhgJ):

Mysql::Error: Got a packet bigger than 'max_allowed_packet' bytes: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465,15466,15467,15468,15469,15470,15471,15475,16252,16256,16257,16258,16259,16260,16263,16267,16268,16269,16270,16272,16279,16282,16283,16284,16285,16286,16287,16302,18042,18043,18044,18045,18046,18062,18063,18064,18065,18066,18067,18068,18084,18085,

Which seems to stem from (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/EADjgREX3Z4J):

And since the 'bad' query starts with: SELECT id FROM `reports` WHERE (`reports`.`id` IN (15462,15463,15464,15465 my guess is its specifically line:

  # reports which have logs entries
      used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).m
  ap(&:id)

https://groups.google.com/forum/?fromgroups#!topic/foreman-users/_Mn4oxXmP7E

Code to functionally get past this error that I've used is (https://groups.google.com/d/msg/foreman-users/_Mn4oxXmP7E/lSKWXpW3JoQJ):

    # used_reports = Report.all(:select => :id, :conditions => {:id => all_reports}).map(&:id)
    used_reports = []
    tmp_arr=all_reports.dup
    slice_size = 10000
    until tmp_arr.empty? do
      first = (tmp_arr.size < slice_size) ? 0 : (tmp_arr.size - slice_size)
      last = tmp_arr.size
      tmp = tmp_arr[first..last]
      used_reports << Report.all(:select => :id, :conditions => {:id => tmp}).map(&:id)
      tmp_arr = tmp_arr - tmp
    end
    used_reports.flatten!.sort


Related issues 2 (0 open2 closed)

Related to Foreman - Bug #8565: report::expire is running very slowlyClosedDominic Cleal12/03/2014Actions
Has duplicate Foreman - Bug #1546: report::expire slow with many reportsDuplicateOri RabinActions
Actions

Also available in: Atom PDF