Project

General

Profile

Actions

Bug #20423

closed

Foreman DB logs index out of range

Added by Chris Baldwin almost 7 years ago. Updated almost 7 years ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
Database
Target version:
-
Difficulty:
easy
Triaged:
Fixed in Releases:
Found in Releases:

Description

Hi,

Why is this a problem?
Our Foreman instances stopped recording new puppet logs in the DB.
What caused the problem?
Our investigation suggests that setting the index in the logs table as an integer (a normal serial instead of bigserial/int8) seems to be the root cause. I'm not 100% sure what code actually does it, my best guess is in here: https://github.com/theforeman/foreman/blob/develop/db/migrate/20101018120621_create_logs.rb , but I don't see a reference to the actual index.
Why fix the problem?
Other long-lived or large installs will run in to this issue. Any table that has significant turnover/growth will face this issue at some point. 2^31-1 isn't a huge number when you have scaled out your infrastructure.
Short term fix?
Clear out logs, reset the index, hope things don't break.
Long term fix?
Change the index to a bigserial/int8.

Writeup:
Per https://groups.google.com/forum/#!topic/foreman-users/hsBX88LEUnY

This is in Foreman 1.12, 1.13, & 1.14. I see the same code in 1.15 for this, but haven't tested it.

After we found that our puppet logs weren't updating, we found (lots of) this in our postgres logs:
2017-07-24 07:03:49 PDT [sess:5974bf14.47da,pid:18394,vitd:9/4891447,tid:0,db:myforemandb]ERROR: integer out of range
2017-07-24 07:03:49 PDT [sess:5974bf14.47da,pid:18394,vitd:9/4891447,tid:0,db:myforemandb]STATEMENT: INSERT INTO "logs" ("message_id", "source_id", "report_id", "level_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5, $6) RETURNING "id"

So we checked the log id #:
myforemandb=> select * from logs order by id desc limit 1;
id | source_id | message_id | report_id | level_id | created_at | updated_at
------------+-----------+------------+-----------+----------+----------------------------+----------------------------
2147483647 | 321 | 1807107 | 292344055 | 2 | 2017-07-24 12:08:01.668757 | 2017-07-24 12:08:01.668757
(1 row)

That id listed is 2^31-1. It turns out that "id" is a signed int, meaning it's 32-bit. I cannot write more logs to the logs table now.

myforemandb=> \d+ logs
Table "public.logs"
Column | Type | Modifiers | Storage | Stats target | Description
------------+-----------------------------+---------------------------------------------------+---------+--------------+-------------
id | integer | not null default nextval('logs_id_seq'::regclass) | plain | |
source_id | integer | | plain | |
message_id | integer | | plain | |
report_id | integer | | plain | |
level_id | integer | | plain | |
created_at | timestamp without time zone | not null | plain | |
updated_at | timestamp without time zone | not null | plain | |
Indexes:
"logs_pkey" PRIMARY KEY, btree (id)
"index_logs_on_level_id" btree (level_id)
"index_logs_on_message_id" btree (message_id)
"index_logs_on_report_id" btree (report_id)
"index_logs_on_source_id" btree (source_id)


Related issues 1 (0 open1 closed)

Is duplicate of Foreman - Bug #13810: Reports fail to upload, Mysql2::Error: Out of range value for column 'id' (logs table)ClosedTimo Goebel02/19/2016Actions
Actions

Also available in: Atom PDF