Tracking issue for char encoding methods

[alexcrichton]

This is a tracking issue for the unstable unicode feature and the char::encode_utf{8,16} methods.

The interfaces here are a little wonky but are done for performance. It's not clear whether these need to be exported or not or if there's a better method to do so through iterators.

[SimonSapin]

How about returning enums like enum OneOrTwo { One(u16), Two(u16, u16) } or enum Utf16Encoding { SingleCodeUnit(u16), SurrogatePair(u16, u16) }?

[alexcrichton]

Certainly possible, but there's also the question of ergonomics here in terms of what to do with that after you've got the information.

[SimonSapin]

I think that one form or another of this functionality that doesn’t require allocation should be exposed. Returning an iterator is nicer than taking &mut [_], but I don’t know about performance.

[nagisa]

I’ve suggested taking &mut [u8; 4]/&mut [u16; 2] and returning usize once. The major downside is inability to convert from slice to array.

Taking anything else than slice also makes following use case not as elegant as it is now:

let mut buffer = Vec::with_capacity(alot);
let mut idx = 0;
loop {
     idx += some_char().encode_utf8(&mut buffer[idx..]).unwrap();
}

[SimonSapin]

How about returning something that both is an iterator and dereferences to a slice?

struct Utf8Char {
    bytes: [u8; 4],
    position: usize,
}

impl Deref for Utf8Char {
    type Target = [u8];
    fn deref(&self) -> &[u8] { &self.bytes[self.position..] }
}

impl Iterator for Utf8Char {
    type Item = u8;
    fn next(&mut self) -> Option<u8> {
        if self.position < self.bytes.len() {
            let byte = self.bytes[self.position];
            self.position += 1;
            Some(byte)
        } else {
            None
        }
    }
}

(“Short” code points have zeros as padding at the start of the array.)

… and similarly for UTF-16, but with [u16; 2] instead of [u8; 4].

[BurntSushi]

@SimonSapin That looks really sweet to me!

[BurntSushi]

@SimonSapin In your deref method, I think that should be &self.bytes[..self.position], right?

[BurntSushi]

@SimonSapin What do you think about also exposing decode_{utf8,utf16} methods? Basically, if you have some bytes and want the next encoded char out of it, today I think you need to decode into a string and then call chars, which is a bit roundabout (and does extra work I believe).