Repository navigation
Conversation
SessionPool only tracked sessions held by an open query result set (the "occupied" map). A session executing a request (an insert, a non-query statement, or a query whose result set has not been returned yet) was in neither that map nor the idle queue. Several places treated the map as if it held every session taken from the pool: - currentOccupiedSize() ignored sessions executing a request, so a saturated write pool reported 0 occupied and 0 available. - setFetchSize, setTimeZone, setEnableRedirection, setEnableQueryRedirection, setVersion and setQueryTimeout did not reach sessions executing a request, which kept the old value after being put back. - close() did not close sessions executing a request; they were later put back into the cleared queue and their connections leaked. "occupied" is now a set of all sessions handed out by getSession() until they are put back, cleaned after a connection error or released by closeResultSet(), and the places above use it. The result-set map is renamed to occupiedByResultSet and maps each session to the result set holding it. closeResultSet() releases a session only if it still belongs to that result set. Before, the second release of a result set (hasNext() auto-releases an exhausted one, the caller's finally releases it again) could put back a session that another thread had meanwhile taken for a new query. It also ignores a wrapper without a session instead of throwing an NPE, and closes the session when closing the operation handle fails instead of leaking its connection.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
TL;DR
SessionPool only tracks sessions held by open result sets (the
occupiedmap), but treats that map as "all sessions in use".As a result,
currentOccupiedSize(), six setters andclose()ignore sessions executing a request.closeResultSet()can also release a session that a later query already has taken, leaks the session when closing fails,and throws an NPE for a wrapper without a session.
All these bugs stem from the same root cause. Each bug has one or more tests in
SessionPoolTestthat fails on master. See table below.Longer Description
Problem
SessionPoolkeeps idle sessions inqueue. Apart from that, it has an additional collection: theoccupiedmap.It contains only sessions whose query result set is still open, until the result set is closed.
A session executing a request – an insert, a non-query statement, or a query whose result set has not been returned yet – is in neither
queuenoroccupied.Note that in the following, I will just refer sessions executing requests instead of spelling out that these requests are inserts, non-query statements, or queries whose result set has not been returned yet.
However, several parts of the class treat
occupiedas if it held every session taken from the pool,which it doesn't:
currentOccupiedSize()ignores sessions executing a request. On a write-heavy pool whose sessions are allwriting, it reports 0 occupied and 0 available. We saw exactly this in production: "occupied" stayed at 0–1 and "available" was 0,
all while writes queued up.
setFetchSize,setTimeZone,setEnableRedirection,setEnableQueryRedirection,setVersionandsetQueryTimeoutapply the new value toqueueandoccupiedonly. A session executing a requestmisses the change and keeps the old value for its whole life after being put back, so the pool ends up with mixed settings.
close()closesqueueandoccupiedonly. A session executing a request at that moment is later putback into the already cleared queue, and its connection is never closed.
closeResultSet()has problems of its own:twice: by
SessionDataSetWrapper.hasNext()once it is exhausted, and again by the caller'scloseResultSet()/close()in afinallyblock. The second call only checks whether thesession is still in
occupied. If another thread took that session in between and opened anew result set with it, the late call puts the session back while the other thread is still
reading, and two threads end up sharing one connection.
session is opened, but the failed session is neither put back nor closed, so its connection
leaks. The failed close also leaves the data set open, so the second release (see 4) sends the
close request again over the broken session and opens yet another replacement for it.
tryConstructNewSession()does not count replacements insize, so the pool can end up withmore sessions than
maxSize.NullPointerException(ConcurrentHashMap.remove(null)).Fix
Only one implementation file was changed:
SessionPool.occupiedis now a concurrent set of all sessions handed out bygetSession(). A session leavesit when it is put back (
putBack), closed after a connection error (closeSession, used by bothcleanSessionAndMayThrowConnectionExceptionoverloads), or released bycloseResultSet().currentOccupiedSize()and the timeout warning reportoccupied.size()(1).close()iterateoccupiedinstead of the result-set sessions (2, 3).The setters only change client-side fields of the session, as they already did for sessions
held by result sets. A session closed by
close()makes its running request fail with aconnection error, the same as reading a result set of a closed pool already did. The retry then
fails with "session pool is closed".
Session.close()returns early when already closed, so asecond close by the request's error handling is harmless.
occupiedmap is renamed tooccupiedByResultSetand maps each session to theSessionDataSetWrapperholding it;occupy(session, wrapper)stores it.closeResultSet()first claims the session withoccupiedByResultSet.remove(session, wrapper)and returns if that fail. So only the first release of this result set closes its operation
handle and hands the session back (4), even if that close failed (5). A failed close goes through
cleanSessionAndMayThrowConnectionException(session), which closes the session and opens areplacement (5). A wrapper without a session is ignored (6).
No public API changes.
Tests
New tests in
SessionPoolTest. Eight of them fail on master, at least one per bug above; the otherfour cover every way a session goes back, so the count cannot drift.
test_CurrentOccupiedSize_WhenInsertIsRunning_CountsItsSessiontest_CurrentOccupiedSize_WhenNonQueryIsRunning_CountsItsSessiontest_Setters_WhenCalledWhileInsertIsRunning_ReachSessionExecutingInserttest_Close_WhenInsertIsRunning_ClosesSessionExecutingInserttest_CloseResultSet_WhenCalledAgainAfterNextQueryTookSession_LeavesItToNextQuerytest_CloseResultSet_WhenClosingOperationHandleFails_ClosesSessionAndStopsCountingIttest_CloseResultSet_WhenCalledAgainAfterFailedClose_DoesNotRetryClosetest_CloseResultSet_WhenWrapperHasNoSession_DoesNothingtest_CurrentOccupiedSize_WhenInsertWasRejected_IsZeroputBacktest_CurrentOccupiedSize_WhenBrokenSessionWasCleanedUp_IsZerotest_CurrentOccupiedSize_WhenResultSetIsOpen_CountsItsSessionUntilClosedcloseResultSettest_CloseResultSet_WhenCalledTwice_PutsSessionBackIntoPoolOnlyOnceThis PR has:
for an unfamiliar reader.
for code coverage.
Key changed/added classes (or packages if there are too many classes) in this PR
SessionPool.java+ tests