Explicitly discourage the use of mutexes inside callbacks (#7792)

The
[specifications](https://opentelemetry.io/docs/specs/otel/metrics/api/#asynchronous-instrument-api)
about callbacks say:

> Callback functions MUST be documented as follows for the end user:
> - **Callback functions SHOULD be reentrant safe.** The SDK expects to
evaluate callbacks for each MetricReader independently.
> - **Callback functions SHOULD NOT take an indefinite amount of time.**
> - Callback functions SHOULD NOT make duplicate observations (more than
one Measurement with the same attributes) across all registered
callbacks.
>
> The resulting behavior when a callback violates any of these
RECOMMENDATIONS is explicitly not specified at the API level.

Currently the Go API mentions that callbacks "needs to complete in a
finite amount of time" and "needs to be concurrent safe", but does not
mention reentrancy.

And while close in meaning, there is a nuance between reentrant and
concurrent safe, and a function locking a plain (non-recursive) mutex is
an example of a function that is concurrent safe while not being
reentrant.

Furthermore, in the Go implementation provided in this repo, the metrics
pipeline locks a mutex (in `produce`) then calls all callbacks, which
means that locking a mutex from within a callback would mean holding 2
mutexes at the same time, which opens the door to a deadlock.

In conclusion, users should be made aware in no uncertain terms that
locking a mutex from callbacks is discouraged and not a supported
behavior

(Relates to the discussion in #7755)

Co-authored-by: Damien Mathieu <42@dmathieu.com>

metric/asyncfloat64.go

@@ -227,7 +227,11 @@ type Float64Observer interface {
 // attributes as another Float64Callbacks also registered for the same
 // instrument.
 //
-// The function needs to be concurrent safe.
+// The function needs to be reentrant and concurrent safe.
+//
+// Note that Go's mutexes are not reentrant, and locking a mutex takes
+// an indefinite amount of time. It is therefore advised to avoid
+// using mutexes inside callbacks.
 type Float64Callback func(context.Context, Float64Observer) error
 
 // Float64ObservableOption applies options to float64 Observer instruments.

metric/asyncint64.go

@@ -225,7 +225,11 @@ type Int64Observer interface {
 // attributes as another Int64Callbacks also registered for the same
 // instrument.
 //
-// The function needs to be concurrent safe.
+// The function needs to be reentrant and concurrent safe.
+//
+// Note that Go's mutexes are not reentrant, and locking a mutex takes
+// an indefinite amount of time. It is therefore advised to avoid
+// using mutexes inside callbacks.
 type Int64Callback func(context.Context, Int64Observer) error
 
 // Int64ObservableOption applies options to int64 Observer instruments.

metric/meter.go

@@ -244,7 +244,11 @@ type Meter interface {
 // Callbacks. Meaning, it should not report measurements for an instrument with
 // the same attributes as another Callback will report.
 //
-// The function needs to be concurrent safe.
+// The function needs to be reentrant and concurrent safe.
+//
+// Note that Go's mutexes are not reentrant, and locking a mutex takes
+// an indefinite amount of time. It is therefore advised to avoid
+// using mutexes inside callbacks.
 type Callback func(context.Context, Observer) error
 
 // Observer records measurements for multiple instruments in a Callback.