1 - 2 and 3 suffered greatly in terms of writing (although I secretly liked 3, but really only for Ben Kingsly)
2 - RDJ's contract is over, Infinity War were the final 2 films he was scheduled for. He might do some cameos, but he's done.
3 - RDJ got too big for his britches. He demanded much more compensation for doing the Marvel movies so over the past few years they've been reducing his role across the board. Also no one wants to work with Gwyneth again either.