Do not call MPI functions directly from finalizers

[giordano]

@vchuravy does this address your concerns in #955 (comment)?

However I'm not happy at all about the implementation, as it adds a new function call before each MPI call. Benchmark:

using MPI
MPI.Init()

const comm = MPI.COMM_WORLD

function bench_rank(n)
    acc = 0
    for _ in 1:n
        acc += MPI.Comm_rank(comm)
    end
    return acc
end

function bench_get_count(n)
    # MPI_Get_count on a fabricated status: another cheap local @mpichk call
    status = MPI.Status(0, 0, 0, 0, 0)
    acc = 0
    for _ in 1:n
        acc += MPI.Get_count(status, UInt8)
    end
    return acc
end

const N = 10_000_000
bench_rank(1); bench_get_count(1)  # compile

for (name, f) in ("Comm_rank" => bench_rank, "Get_count" => bench_get_count)
    best = minimum((@elapsed f(N)) for _ in 1:5)
    println(name, ": ", round(best / N * 1e9, digits=2), " ns/call")
end

# allocations on the hot path must be zero
alloc = @allocated bench_rank(1000)
println("allocations for 1000 calls: ", alloc, " bytes")

MPI.Finalize()

on master, with 2 ranks:

Comm_rank: 3.88 ns/call
Comm_rank: 3.88 ns/call
Get_count: 5.56 ns/call
Get_count: 5.59 ns/call
allocations for 1000 calls: 0 bytes
allocations for 1000 calls: 0 bytes

on this branch:

Comm_rank: 5.5 ns/call
Comm_rank: 5.5 ns/call
Get_count: 15.7 ns/call
Get_count: 15.77 ns/call
allocations for 1000 calls: 0 bytes
allocations for 1000 calls: 0 bytes

The overhead is non-negligible (although of the order of ~nanoseconds)

Finalizers can run at any point of the program where the garbage
collector is invoked, including concurrently with an MPI call made by
another thread (which is permitted only when MPI was initialized with
MPI_THREAD_MULTIPLE), or in the middle of an MPI call which runs a
Julia callback, such as a reduction with a user-defined operator
(calling MPI functions inside such callbacks is erroneous).

Instead of calling the corresponding MPI_*_free function directly,
the finalizers of MPI handle objects (communicators, groups, datatypes,
operators, infos, requests) now capture the raw handle value in a
closure and enqueue it, and the queue is drained before the next MPI
call made with @mpichk: the fast path of the drain is a single atomic
counter check, and enqueueing from a finalizer uses trylock, followed
by re-registration of the finalizer when the lock is contended, as
finalizers must never block on a lock. This design is similar to how
PythonCall.jl defers the freeing of Python objects from finalizers to
the next time the GIL is held.

Explicit calls to MPI.free keep freeing the handle immediately. The
finalizers of Win and FileHandle objects are unchanged: freeing
those handles is a synchronizing/collective operation, which deferral
would not make safe.