lua: fix likely double-frees and use of undefined pointer state
Split from !1085 (closed).
the pointer *pp_data
, provided by the callers of these callback
functions, is left in an undefined state upon failed asprintf()
.
the returned success state is VLC_SUCCESS
in such a circumstance, and
along with questionable reliability of whether or not *pi_data
equals 0
only under such a condition (consider the use of vlclua_todata()
), means
that it may in fact be impossible for calling code to determine whether or
not it is safe to use their provided data pointer variable after execution
of the callback. it is expected that calling code (difficult to track down
due to the obfuscation of a callback mechanism) is likely misusing this
unknown pointer state in such a failure condition, for instance via
unconditional free()
.
resetting the pointer to null upon asprintf()
failure eliminates the leak
of unknown state, and assuming that calling code likely is modelled such
that it initialises the variable to null, then upon return of VLC_SUCCESS
from callback execution, unconditionally uses and frees the pointer, this
should now work correctly in the scenario of asprintf()
having failed.
furthermore, the callback functions initially allocate memory to *pp_data
via vlclua_todata()
, then conditionally free()
it before making an
asprintf()
call with pp_data
. without the null reset, upon asprintf()
failure, if the asprintf()
implementation does not write to the pointer
variable, and if this leads to the callback caller as above to free()
the
pointer variable, then free()
will have been called twice upon the same
allocation (a "double-free"). resetting the pointer to null upon failed
asprintf()
additionally fixes this issue. note that this had to be done
in an additional third location in order to fully address the double-free
bug.