Why would enabling keepalives in Nginx increase my response time?

Why would enabling keepalives in Nginx increase my response time?

I've got nginx servers behind an Amazon ELB, and recently began testing
enabling keepalives between the nginx server and ELB. Once I enabled
keepalive, I saw response times double from 25ms to 50ms. I'd expect them
to decrease since the connection is staying open and theres less overhead
for the handshake and new socket opening.
All other server metrics improved with enabling keepalive, number of tcp
timeouts decreased, CPU usage decreased, but response time took a hit.
I tried disabling postpone_output as I thought maybe theres some sort of
buffering issue going on, but that had no effect.
Some additional info. The nginx servers are proxying to upstreams, when
graphing the upstream response time, it appears that the upstream response
is the value thats actually increasing when keep alives get enabled
between the ELB and Nginx which is affecting the overall response time.