Are those popular APIs?

I've found Copilot useful when writing greenfield code, but very unhelpful generating code that uses APIs not popular enough to have significant coverage on StackOverflow. Even if I have examples of correct usage in the same file it still guesses plausible but wrong types.

I haven't bought GPT 4 but I'm curious if it's much better at this.

If you don't mention a library by name it is liable to make something up by picking a popular library in another language and converting the syntax to the language you asked for.

If you ask for something impossible in a library it will also frequently make up functions or application settings. If you ask for something obscure but hard to do, it might reply that it's impossible but it is possible if you know how and teach it.

I sort of compare prompt engineering to Googling - you sometimes have to search for exactly the right terms that you want to appear in the result in order to get the answer you're looking for. It's just that the flexibility of ChatGPT in writing a direct response sometimes means it will completely make up an answer.

There's also a limitation that the web interface doesn't actually let you upload files and has a length limit for inputs. For Copilot, I'm looking forward to Copilot X: https://www.youtube.com/watch?v=3surPGP7_4o

This was neither. I've forgotten the exact words I typed but it was something like this.

Prompt:

    fn encode(value: Foo) {
        capnproto::serialize_packed:: serialize_message(value);
    }

    fn decode(input: &[u8]) {

Expected:

   capnproto::serialize_packed:: deserialize_message(input);

Generated

     capnproto::PackedMessageDeserializer::deserialize(input)

Maybe a difference between prompts or v3.5 vs v4. I asked ChatGPT v4 and it gave an answer that used capnproto::serialize_packed::read_message. I asked it about the difference between read_message and deserialize_message and this is the response it gave (truncated, as it got some parts wrong):

> In summary, you should use read_message when working with packed messages and deserialize_message when working with unpacked messages. Make sure you choose the appropriate serialization and deserialization functions based on the format in which your messages are stored or transmitted.

Google was surprisingly little help on the topic. At best it pointed me to https://capnproto.org/encoding.html#packing which covers the general idea but glosses over parts.

The problem with ChatGPT is you can never be certain or confident that its answers are correct. It’s as useful as rolling dice in guessing a number sometimes. With the training from the internet, the dice are loaded, but the answer is still likely to be wrong because it just assembles words together.