The problem
The stack is Gin, GORM v2, pgx/v5 in simple protocol mode, PostgreSQL, and PgBouncer. The scenario: when a customer places an order, the item is reserved for 30 minutes. Once the reservation expires, it is released and the item is removed from the cart.
Every five minutes, a background job finds expired reservations and releases them. Then support receives a ticket:
A customer places an order, the frontend shows a “reservation expires in 30 minutes” timer, and five minutes later the item vanishes from the cart. It affects every customer.
Getting the vocabulary straight
Time handling gets confusing quickly because everyone says “time” while meaning something different. Let us define the terms before looking at the code. This diagram explains more than the definitions alone: the same event represented in three different ways.
the same event
│
┌─────────────────────┼────────────────────────┐
▼ ▼ ▼
PostgreSQL PostgreSQL Go
timestamp timestamptz time.Time
date and time an instant, an instant plus
with no zone: stored in UTC a location for display
'2026-09-17 13:06'
IS THAT 13:06 13:06+03 and 10:06+00 the same value prints
IN MOSCOW OR UTC? are the same instant. as 16:06 (+03)
NOBODY KNOWS. The zone is not or as 13:06 (UTC).
Only the writer stored, output takes
knows. the session time zone.To recap:
timestamp without time zone: stores a date and time with no zone, such as 2026-09-17 13:06:39. It does not identify an instant; it only says that a clock read 13:06. Only the writer knows which time zone that clock used. All application-level timestamp columns in this project use this type: created_at, reserved_at, reserved_until, and deleted_at.
timestamptz: an instant. PostgreSQL stores it in UTC and converts it to the session time zone on output, so the same instant may be displayed differently on different connections.
time.Time (Go): an instant together with a location used for display. Go can print the same instant as 14:00 in UTC+03 or as 11:00 in UTC. time.Now() uses the host’s local time zone; time.Now().UTC() returns the same instant in UTC.
NOW() (SQL): a timestamptz representing the start of the current transaction. Its value does not change during a long transaction; clock_timestamp() returns the actual current time.
Session time zone: the value of the TimeZone parameter for a particular database connection. A session is one connection. When the application opens a connection, PostgreSQL creates a session for it and keeps that session alive until the connection closes. The application pool therefore has one session per open connection, and each session receives its own time-zone setting. The client can request a time zone during startup; otherwise PostgreSQL applies a default, usually from the server configuration. Database-level (ALTER DATABASE ... SET TimeZone) and role-level (ALTER ROLE ... SET TimeZone) settings can override that default. You can inspect the effective value only from inside the session by running SHOW timezone on the same connection. This is why the result in psql tells you nothing about the time zone used by the application’s connections.
Act 1: angry customers, a reservation gone in five minutes
The most obvious suspect is the background job. It runs every five minutes, and the reservation in the ticket lasted about five minutes. Coincidence? I think not.
We inspect the database and find a reservation that was released too early:
-[ RECORD 1 ]--------+-------------------------------------
id | 01a0af78-0228-7964-8afa-a79325bc757e
reserved_at | 2026-09-17 13:06:39.282671
reserved_until | 2026-09-17 13:36:39.282671
released_at | 2026-09-17 13:11:02.516204
created_at | 2026-09-17 16:04:32.804142
prepared_at | 2026-09-17 16:06:21.090645
deleted_at | 2026-09-17 13:20:34.397342What stands out immediately:
reserved_untilequalsreserved_at + 30 minutes, down to the microsecond. The deadline is computed correctly: the application writes “start plus 30 minutes,” not “plus 5.”released_atis twenty-five minutes earlier thanreserved_until. The job records the time when it released the reservation, so the reservation was closed long before its deadline. The support ticket is now visible in the data. One more detail stands out: only about five minutes passed betweenreserved_atandreleased_at.deleted_atis earlier thancreated_at. The cart appears to have been deleted before it was created: deletion in the past, creation in the future. It looks like a time machine.
Plot the values on a timeline exactly as they appear in the database, and the time machine becomes obvious:
time in the DB: 13:06 13:20 13:36 16:04 16:06
│ │ │ │ │
cart created │ │ │ ● │ created_at
cart prepared │ │ │ │ ● prepared_at
reservation started ● │ │ │ │ reserved_at
reservation expires │ │ ● │ │ reserved_until
cart deleted │ ● │ │ │ deleted_at
events in a sensible order, timestamps all over the placeThe deadline is correct, and the background job released the reservation. But why did a fresh reservation appear in the “expired” result set? Let us inspect the query:
// repositories/reservation_repo.go: selecting expired reservations
var reservationIDs []uuid.UUID
err := r.dbFromCtx(ctx).WithContext(ctx).
Model(&models.Reservation{}).
Where("reserved_until IS NOT NULL").
Where("released_at IS NULL").
Where("reserved_until <= NOW() - interval '5 minutes'").
Order("reserved_until ASC").
Limit(limit).
Pluck("id", &reservationIDs).ErrorThe query compares a column with NOW(). Everything depends on how production evaluates that expression.
Act 2: NOW() answers in Moscow time
Let us look at the row again. Its columns fall into two groups:
13:06 / 13:36 / 13:11 / 13:20 reserved_at, reserved_until, released_at, deleted_at
16:04 / 16:06 created_at, prepared_atThe second group is exactly three hours ahead. What do these +03:00 columns have in common? They are populated in the same way:
WHO WRITES THE COLUMN?
application (Go) server (PostgreSQL)
pgx → .UTC() → literal DEFAULT now() / gorm.Expr("NOW()")
│ │
▼ ▼
┌──────────────────┐ ┌──────────────────┐
│ UTC time │ │ SESSION-LOCAL │
│ │ │ time │
│ reserved_at │ │ created_at │
│ reserved_until │ │ prepared_at │
│ released_at │ │ │
│ deleted_at │ │ │
└──────────────────┘ └──────────────────┘
│ │
└──────────────┬─────────────────────┘
▼
ONE row, TWO different clocks// created_at: a database-side default. GORM leaves the column out of the INSERT,
// so PostgreSQL supplies the value.
type Reservation struct {
ID uuid.UUID `gorm:"type:uuid;primaryKey;default:gen_random_uuid()"`
ReservedAt time.Time `gorm:"column:reserved_at;not null"`
ReservedUntil time.Time `gorm:"column:reserved_until;not null"`
ReleasedAt *time.Time `gorm:"column:released_at"`
DeletedAt *time.Time `gorm:"column:deleted_at"`
CreatedAt time.Time `gorm:"column:created_at;not null;default:now()"`
}
// prepared_at: an explicit NOW() in SQL
if err := tx.Model(&models.Reservation{}).
Where("id = ?", reservationID).
Update("prepared_at", gorm.Expr("NOW()")).Error; err != nil {
return err
}Both DEFAULT now() and gorm.Expr("NOW()") are evaluated by PostgreSQL and converted using the session time zone. So which time zone did the production session use?
shop=> SELECT now();
now
-------------------------------
2026-09-17 16:11:07.85158+03Moscow time, UTC+03:00. Now for the important part. now() returns a timestamptz, which represents an instant. To store it in a timestamp column or compare it with one, PostgreSQL must convert it to a value with no time zone. It performs that conversion using the session time zone. The instant is unchanged, but it is rendered as Moscow local time:
shop=> SELECT now()::timestamp;
now
--------------------------
2026-09-17 16:11:23.232007 ← the same instant, rendered in Moscow timeThe conversion looks like this: one instant produces different wall-clock values depending on the time zone used to render it.
a point in time (timestamptz)
│
┌───────────────┴───────────────┐
▼ ▼
session time zone UTC session time zone +03
│ │
▼ ▼
timestamp '13:06' timestamp '16:06'
UTC time Moscow time
═══ one instant, two different wall-clock values ═══
the session time zone determines how the instant is renderedNow we can reconstruct the incident. The reservation started at 13:06:39 UTC, the same instant as 16:06:39 in Moscow. The job runs every five minutes, so consider its first run after that, at roughly 16:11 Moscow time. This is how the filter evaluates:
WHERE reserved_until <= NOW() - interval '5 minutes'
NOW() as local time: 16:11 (Moscow, rendered using the session time zone)
cutoff: 16:06 (16:11 minus 5 minutes)
reserved_until in DB: 13:36 (written by Go as a UTC wall-clock value)13:36 is less than or equal to 16:06, so according to this comparison the reservation expired hours ago, even though it is only minutes old. The job selected it, released the reservation, and removed the item from the cart.
The five minutes mentioned in the ticket have nothing to do with the timer or with reserved_until. Five minutes is the job’s polling interval. The reservation was released on the first run after its creation, which is why the customer never saw it last longer than five minutes.
The same row, normalized to one time zone
Now we can reinterpret the strange values from Act 1. Once every column is expressed in UTC, the time machine disappears and an ordinary sequence of events emerges:
13:04:32 UTC created_at (16:04:32 MSK, server DEFAULT) cart created
13:06:21 UTC prepared_at (16:06:21 MSK, server NOW()) cart prepared
13:06:39 UTC reserved_at (written by Go in UTC) reservation started
13:11:02 UTC released_at (written by Go in UTC) reservation released by the job
13:20:34 UTC deleted_at (written by Go in UTC) cart clearedThe cart was prepared 18 seconds before the reservation started, so that part is correct. deleted_at appeared to be earlier than created_at only because the row mixed wall-clock values from two time zones. In reality, the customer returned to an empty cart 16 minutes after it was created and about nine minutes after the job released the reservation. There was no time machine, only two time zones in one row.
Act 3: who writes the time, and in which zone
Let us trace how values reach the timestamp columns and which time zone each layer uses:
Go model (time.Time)
│
▼
GORM passes the value through, builds SQL, skips DEFAULT columns
│
▼
pgx encodes the value as query text; UTC appears here
│
▼
PostgreSQL drops the offset from the literal and evaluates NOW(),
DEFAULT, and comparisons using the session time zoneGORM passes the supplied time.Time through without converting it. pgx handles the encoding. This gives us three write paths.
Source 1: Go parameters are written as UTC
Here is the code that creates a reservation:
// use case: a 30-minute reservation
now := time.Now()
reservation := models.Reservation{
ReservedAt: now,
ReservedUntil: now.Add(30 * time.Minute),
}
if err := db.WithContext(ctx).Create(&reservation).Error; err != nil {
return err
}There is no explicit time-zone handling here. Whatever time.Now() returns is passed to the database. The important behavior appears further down the stack.
The project uses simple protocol mode. Instead of sending parameters separately, pgx interpolates their values into the SQL string before sending the query to PostgreSQL. Because the value enters the query before the server can match it to a column, pgx selects an encoder from the Go type. It maps time.Time to timestamptz and serializes the instant in UTC.
Let us follow one value through the entire path. Consider a host with TZ=Europe/Moscow, where time.Now() returns Moscow local time:
Go now = 2026-09-17 16:06:39.282671 +0300 MSK
│
▼ GORM takes the value as is and puts it in the INSERT
pgx '2026-09-17 13:06:39.282671+00' ← .UTC(), a literal in the SQL string
│
▼ INSERT INTO reservations (reserved_at, ...)
VALUES ('2026-09-17 13:06:39.282671+00', ...)
PostgreSQL reserved_at is a timestamp column: the offset is dropped,
the time is not converted
│
▼
in the table 2026-09-17 13:06:39.282671The last step explains why a value created in Moscow time ends up as a UTC wall-clock value in the table. We can verify that behavior directly in PostgreSQL:
shop=> SELECT '2026-09-17 13:06:39+00'::timestamp;
2026-09-17 13:06:39The result: Go parameters reach the database as UTC wall-clock values regardless of the location attached to the original time.Time, whether it came from time.Now() on a host in UTC+03 or from an explicit .UTC() call. Our code does not normalize the value itself; this behavior follows from how pgx encodes time.Time in simple protocol mode.
Source 2: DEFAULT now() and the session time zone
GORM leaves fields tagged with default:now() out of the column list in an INSERT, allowing PostgreSQL to supply the value. The resulting wall-clock time comes from the session time zone: a UTC session writes UTC, while a UTC+03 session writes Moscow local time. That is where the row’s created_at value came from.
Source 3: gorm.Expr(“NOW()”) and the session time zone
An explicit NOW() uses the same server-side mechanism: PostgreSQL converts the timestamptz instant to a local wall-clock value when storing it in a timestamp column. This project has more than a dozen such cases, including ordinary UPDATE statements and assignments in an upsert’s ON CONFLICT clause.
The most dangerous case is a comparison between a timestamp column and NOW(). The column contains a wall-clock value with no zone. Before PostgreSQL can compare it with an instant, it must decide which instant that value represents. It uses the session time zone to make that decision:
WHERE reserved_until <= NOW()
reserved_until = '13:36'
│
"Which time zone is 13:36 in?"
│
┌───────────────┴───────────────┐
▼ ▼
session time zone: UTC session time zone: +03
13:36 means 13:36 UTC 13:36 means 13:36 MSK
NOW() is 13:06 UTC NOW() is 16:06 MSK
│ │
▼ ▼
13:36 UTC > 13:06 UTC 13:36 MSK ≤ 16:06 MSK
│ │
▼ ▼
reservation stays active reservation is releasedThe row and SQL are identical in both cases. The session time zone determines which zone PostgreSQL assigns to the bare timestamp, and therefore determines the result.
Summary
Here are all three sources and the values they produce:
the timestamp column
▲
┌───────────────────┼───────────────────┐
│ │ │
SOURCE 1 SOURCE 2 SOURCE 3
Go parameters DEFAULT now() NOW() in SQL
│ │ │
pgx: .UTC() server, session server, session
'13:06+00' time zone time zone
│ │ │
▼ ▼ ▼
UTC wall time session-local session-local
(always) time (INSERT) time (UPDATE)
│ │ │
└─────────┬─────────┴───────────────────┘
▼
session UTC → all three write the same thing ✓
session +03 → two clocks in one row ✗
│
▼
NOW() comparisons use the session time zone too → shift by the offsetHere is the same comparison in table form:
| Operation | Time zone of the stored value | What it depends on |
|---|---|---|
Writing time.Time into timestamp in SimpleProtocol or Exec mode |
UTC wall time | pgx encodes from the Go type and calls .UTC() |
Writing time.Time into timestamp in a describe-based mode |
the location attached to the value | pgx encodes from the OID reported by the server |
DEFAULT now() and gorm.Expr("NOW()") |
the session time zone | the session time zone |
Comparing timestamp with NOW() |
the column is interpreted in the session time zone | the session time zone |
As long as the session uses UTC and the application uses simple protocol mode, all three paths write compatible values. With a non-UTC session, the table mixes two clocks and every comparison with NOW() shifts by the zone offset. In production, the session used UTC+03, Go values were written as UTC, and NOW() was converted to Moscow local time.
Why staging and CI stayed green
The release pipeline looked like this:
docker CI stage prod
(UTC) (UTC) (UTC) (MSK)
│ │ │ │
▼ ▼ ▼ ▼
green green green BOOM!
└──────────┴───────────┘ │
all quiet │
▼
the same code,
the only difference is the time zoneStaging and production ran identical code; the different behavior came from a single configuration value: the server’s default time zone. Staging passed and the release went to production. Where SHOW timezone returned UTC, the job correctly waited for the reservation to expire. Where it returned MSK, the job released the reservation almost immediately.
The mistake was not in the deadline calculation. It was an implicit contract between the application and its environment. Staging proved only that the code worked when the session time zone was UTC. The unwritten assumption was: “The application writes UTC wall-clock values into timestamp columns, so NOW() must produce compatible UTC values.” Nothing enforced that assumption. Staging happened to satisfy it; production exposed it.
A practical rule follows: if behavior depends on an environment setting, that setting is part of the application’s contract. Enforce it in code or verify it in every environment.
The existing tests could not catch the problem. The developer’s database runs in Docker, CI has a separate database, and both use UTC. In those sessions, NOW() produces values compatible with the UTC values written by the application, so the comparisons work and every test stays green. The discrepancy appears only when SHOW timezone returns a non-UTC value. Reproducing it requires deliberately connecting a test to a database with a non-UTC time zone. It is a classic trap: the tests cover the logic, but the failure lives in configuration.
The fix we shipped
We considered three options:
- Set the connection time zone to UTC.
- Change the time zone of the database servers. That is DBA territory. Guaranteeing UTC in Docker, CI, two staging environments, and production adds another setting that must be configured correctly in every new environment.
- Migrate the columns to
timestamptz. That requires migrating every application table and auditing every time comparison before release. It is expensive and risky.
We chose the first option. It is fast, inexpensive, fixes the issue in every environment at once, and requires no migration.
This is treatment, not a cure. Every affected column is populated from either now() or time.Now(), and both represent an instant, which belongs in timestamptz. Storing that instant in timestamp discards its time zone. Enforcing UTC preserves the historical contract that “our timestamp columns contain UTC wall-clock values,” but it protects only connections opened through this application. Any client that bypasses this connection setup can write values in a different time zone. The long-term fix is a type migration, which remains in the backlog by choice rather than being abandoned.
Enforcing UTC in the startup parameters
func Connect(dsn string) *gorm.DB {
syncOnce.Do(func() {
pgxConfig, err := pgx.ParseConfig(dsn)
if err != nil {
log.Fatalf("failed to parse postgres config: %v", err)
}
pgxConfig.DefaultQueryExecMode = pgx.QueryExecModeSimpleProtocol
pgxConfig.RuntimeParams["TimeZone"] = "UTC"
...pgx sends this parameter in the startup packet, so every new connection requests UTC rather than configuring one arbitrary pooled connection. PgBouncer tracks TimeZone and applies it to the server connections assigned to that client. Adding ?TimeZone=UTC to the connection string produces the same result.
PgBouncer will not drop this parameter. TimeZone is one of the parameters it tracks by default, alongside client_encoding and standard_conforming_strings, and it applies those values to the server connection itself.
Fail fast
The line RuntimeParams["TimeZone"] = "UTC" expresses intent, not a guarantee. It describes how we build the configuration but does not prove which value became effective at the other end.
The setting can disappear inside the application. Someone may remove the line during a refactor or merge, or another proxy in front of the database may fail to forward it. Neither case produces an error. The connection simply receives a different time zone.
The existing environments will not expose this mistake on their own. Docker and CI already default to UTC, so removing the line changes nothing there. The problem appears only in production, where the default is different. In other words, the setting protects us from an environmental difference, but that protection is tested only where failure is most expensive. We therefore verify the effective value at startup:
connectionTimeZone, err := readConnectionTimeZone(sqlDB)
if err != nil {
log.Fatalf("failed to read database connection time zone: %v", err)
}
if err := validateTimeZone(connectionTimeZone); err != nil {
log.Fatalf("database connection check failed: %v", err)
}func validateTimeZone(value string) error {
if !strings.EqualFold(value, "UTC") {
return fmt.Errorf("session time zone is %q; want \"UTC\": "+
"the connection did not apply the requested time zone "+
"(check the connection string, the pooler, and the server defaults)", value)
}
return nil
}Etc/UTC is equivalent to UTC, but it is not the value we requested. If a different name comes back, our startup parameter may not be the source of the effective setting. Today the server default may be UTC-equivalent; tomorrow someone may change it and put us back in Moscow time.
This check also has limits. It runs SHOW timezone on one connection returned by the pool, not on every connection. If every new connection must be verified, move the check to pgxConfig.AfterConnect, at the cost of one extra round trip per connection. The startup check runs only once, so it will not detect a later SET timezone within a session. It also says nothing about connections created outside this function. A background worker with its own pool or a one-off script may bypass Connect, leaving it without either the UTC setting or the check. Finally, an answer of UTC does not prove that our parameter arrived: the server may already use UTC by default. For data safety, the source of the value matters less than the effective time zone. A container that fails at startup is visible in monitoring; silently corrupted data may go unnoticed.
The entire path from configuration to an active session looks like this:
pgxConfig.RuntimeParams["TimeZone"] = "UTC"
│ intent
▼
sent with every new connection
│
▼
what zone does the session really have?
│
┌─────────────┴─────────────┐
│ UTC │ another time zone
▼ ▼
the server session the UTC setting did not apply,
uses UTC so the session kept another zone
│ │
▼ ▼
SHOW timezone = "UTC" SHOW timezone ≠ "UTC"
│ │
▼ ▼
startup continues ✓ log.Fatalf at startup
(the crash is visible right away)Tests mirror production
The test connection gets the same guarantee through the DSN:
func ensureDSNTimeZone(dsn string) string {
if !strings.Contains(dsn, "://") {
return dsn
}
if i := strings.Index(dsn, "?"); i >= 0 {
if values, err := url.ParseQuery(dsn[i+1:]); err == nil {
for key := range values {
if strings.EqualFold(key, "timezone") {
values.Del(key)
}
}
values.Set("TimeZone", "UTC")
return dsn[:i] + "?" + values.Encode()
}
return dsn + "&TimeZone=UTC"
}
return dsn + "?TimeZone=UTC"
}A separate test verifies the session time zone of the test connection. If someone breaks ensureDSNTimeZone or changes the DSN format, the test catches the problem before production does.
The function has two limitations. First, it understands only the URL form of a DSN. A string such as host=db.internal user=shop dbname=shop is returned unchanged, leaving the test connection without the UTC setting. Second, the function overwrites every time zone in the DSN, including one set intentionally. A test cannot use it to connect with a non-UTC time zone because ?TimeZone=Europe/Moscow becomes UTC. Such a test must open its own connection outside the shared pool.
After the fix
-[ RECORD 1 ]--------+-------------------------------------
reserved_at | 2026-09-18 08:02:32.021884
reserved_until | 2026-09-18 08:32:32.021884
released_at | 2026-09-18 08:35:46.328199
created_at | 2026-09-18 08:00:28.703221
prepared_at | 2026-09-18 08:02:10.986437
deleted_at | 2026-09-18 08:44:12.517903Every column now uses the same clock. deleted_at no longer appears earlier than created_at, reservations last the full thirty minutes, and released_at is three minutes later than reserved_until. Those three minutes are the delay before the first job run after the actual expiration time.
All together
Here is the complete connection function with the UTC setting, pool configuration, and startup check:
func Connect(dsn string) *gorm.DB {
syncOnce.Do(func() {
pgxConfig, err := pgx.ParseConfig(dsn)
if err != nil {
log.Fatalf("failed to parse postgres config: %v", err)
}
pgxConfig.DefaultQueryExecMode = pgx.QueryExecModeSimpleProtocol
pgxConfig.RuntimeParams["TimeZone"] = "UTC"
registeredDSN := stdlib.RegisterConnConfig(pgxConfig)
db, err = gorm.Open(
postgres.New(postgres.Config{
DSN: registeredDSN,
DriverName: "pgx",
}),
&gorm.Config{
PrepareStmt: false,
TranslateError: true,
},
)
if err != nil {
log.Fatalf("failed to connect database: %v", err)
}
sqlDB, err := db.DB()
if err != nil {
log.Fatalf("failed to get sql.DB from gorm.DB: %v", err)
}
sqlDB.SetMaxOpenConns(20)
sqlDB.SetMaxIdleConns(10)
sqlDB.SetConnMaxLifetime(15 * time.Minute)
sqlDB.SetConnMaxIdleTime(5 * time.Minute)
if err := sqlDB.Ping(); err != nil {
log.Fatalf("failed to ping database: %v", err)
}
var zone string
if err := sqlDB.QueryRow("SHOW timezone").Scan(&zone); err != nil {
log.Fatalf("failed to read database connection time zone: %v", err)
}
if !strings.EqualFold(strings.TrimSpace(zone), "UTC") {
log.Fatalf("session time zone is %q; want \"UTC\": "+
"the connection did not apply the requested time zone "+
"(check the connection string, the pooler, and the server defaults)", zone)
}
})
return db
}How to debug this
Use this checklist when you suspect a time-zone mismatch:
- Run
SHOW timezonein every environment from the application and through the same pool it normally uses, not only from psql. A psql session may use a different time zone. If the value differs from the server default, keep investigating: database-level and role-level settings may override the default, and an application setting overrides them in turn. - Look for columns from the future:
SELECT created_at, reserved_at FROM ... WHERE created_at > reserved_at. Rows where server-generated timestamps are ahead of client-written values reveal mixed time zones immediately. - Compare values by source: compare columns written by Go with columns written by
DEFAULT now()orNOW(). The difference should be consistent across affected rows and should match the session time zone offset. It is often a whole number of hours, although offsets such as+05:30and+05:45also exist. - Test with a non-UTC time zone: connect to the test database with
TimeZone=Europe/MoscoworPGTZ=Europe/Moscow, then run the time-related tests. If they still pass, the logic is genuinely independent of the session time zone.
Gotchas
The UTC setting applies only to connections created here. A new service, a one-off script, a migration, or an analyst in psql opens a separate connection and knows nothing about this configuration. No error is raised; the new connection may simply use a different time zone. This is why the startup check belongs in the connection code: it verifies the connection the application actually uses, not the database as a whole.
One column can have multiple writers. During an ordinary .Update(), GORM fills updated_at with a Go time value, which pgx writes as UTC. If one upsert writes gorm.Expr("NOW()") to the same column, that code path starts using server time instead. The value in a particular row therefore depends on which path wrote it last.
timestamptz values are rendered in the session time zone. The task framework tables return text using the session time zone, but the underlying instant remains correct; only its representation changes. After enforcing UTC, the output becomes consistent. Before that, this explains why framework tables may show +00 while application tables contain a bare local time.
Takeaways
timestampstores a wall-clock value with no time zone. The writer determines which zone that value represents. Go parameters are written as UTC, whileNOW()andDEFAULT now()are converted using the session time zone. A non-UTC session therefore creates two clocks in one row.- In this incident, the session time zone affects three places: the value written by
NOW(), the interpretation of atimestampduring comparison, and the way atimestamptzvalue is rendered. Those are the three points to inspect. - The same code behaved differently in staging and production because of an unenforced assumption. The application expected the database session to use UTC but never required it. Assumptions like this must be enforced in code or verified in every environment.
- The existing tests did not catch the problem because they covered the logic while the failure lived in configuration. A test run with a non-UTC time zone, together with the same UTC setting on ordinary test connections, covers both cases.
- The startup parameter is still a request rather than proof. The startup check verifies the effective value: if
SHOW timezonereturns anything other thanUTC, the application refuses to start. - Enforcing UTC protects the application’s connections, not the data itself. As long as the columns remain
timestamp, the rule that “these values are UTC” depends on every writer following the same convention, including other services, migrations, and analysts using psql.
This failure can no longer remain hidden. A changed server default, a new environment, or a connection created outside the shared configuration produces the same visible result: the application refuses to start, and the first log message reports which time zone it received instead of UTC. The check runs at every launch in every environment, including environments that do not exist yet.
The difference is in the cost. A silent mismatch cost us prematurely released reservations, a support ticket, and a day of investigation. At worst, a failed startup costs us one deployment.
References
- Date and time types in PostgreSQL.
- The TimeZone parameter and its defaults.
- Query execution modes in pgx.
- GHSA-j88v-2chj-qfwx: injection in the simple protocol before pgx v5.9.2.
- Startup parameters in the PgBouncer configuration.
- How GORM populates timestamp fields.
My Telegram channel: @tsymbaldev.