Fix infinite loop on corrupt bitstreams
Change ENOMEM into EINVAL, since at this point memory allocation errors don't occur, and bitstream decoding errors are not fatal. Then, unref data after decoding failure to prevent re-entering the loop with the same data.
(See sample from #277 (closed).)
Edited by Ronald S. Bultje