I uploaded a 92k word, unpublished sci book that I wrote to GPT4, then asked it basic plot questions. Who is the protagonist, did this character die, etc? The LLM usually failed, I think because fiction writing tends to imply rather than state important points. It even had trouble identifying dialogue because it didn't identify smart quotes as quotation marks - until I corrected the model. I'd like to see a larger test on understanding fiction.
I uploaded a 92k word, unpublished sci book that I wrote to GPT4, then asked it basic plot questions. Who is the protagonist, did this character die, etc? The LLM usually failed, I think because fiction writing tends to imply rather than state important points. It even had trouble identifying dialogue because it didn't identify smart quotes as quotation marks - until I corrected the model. I'd like to see a larger test on understanding fiction.