Vectorize loops with exit users with tail folding

[lukel97]

https://godbolt.org/z/c7daf875K

If a loop has users in the exit block, we currently can't vectorize it with EVL tail folding and instead fall back to a scalar epilogue:

define i32 @f(ptr %p, ptr %q, i32 %n) {
entry:
  br label %loop

loop:
  %iv = phi i32 [0, %entry], [%iv.next, %loop]
  %gep = getelementptr i32, ptr %p, i32 %iv
  %x = load i32, ptr %gep
  %y = add i32 %x, 1
  store i32 %y, ptr %gep
  %iv.next = add i32 %iv, 1
  %done = icmp eq i32 %iv.next, %n
  br i1 %done, label %exit, label %loop

exit:
  ret i32 %y
}

With EVL I think it should be possible to form a VPInstruction::ExtractLastElement which reads from the last active lane using the EVL from get-active-vector-length.

It should also be possible to do this with regular mask tail folding via @llvm.ctpop

[preames]

For RISCV at least, doing this for mask predication is also straight forward with vcpop.m. Since you know the one bits are a prefix of the mask, the pop count also gives you the last active index.

[lukel97]

For RISCV at least, doing this for mask predication is also straight forward with vcpop.m. Since you know the one bits are a prefix of the mask, the pop count also gives you the last active index.

Oh good point. So we could probably support this generically for any sort of tail folding, I'll update the issue.

[fhahn]

I think with #148817, we should have all that is needed to handle this in the general case: #149042