THE STORY OF THE FILM SO FAR:
Intrepid adventurer Joe has voyaged into the confusing world of AI use and come back with a set of guidelines he follows when using AI. Ignoring the question of whether to use AI, he shares his map for getting through the AI jungle.
And here’s some more!
1. AI will suck up to you. Getting honest feedback from it is harder than you think.
One of the first things I tried with AI was have it give feedback on a screenplay. I found it useful. Not as good as the best feedback I’ve gotten from humans, but not as bad as the worst. So middling useful, as far as feedback goes, and useful in the same way as human feedback: not something to take as gospel, but a spur to help my thinking.
But unless you’re careful, AI will always give you the most positive gloss. Because it wants to make you feel good.
Sometimes that’s nice. We all need positive reinforcement sometimes, and AI is always willing to provide it.
But if you want real honest feedback, you’ve got to carefully craft your prompt. And do your best to keep your wishes from creeping into it. Because AI will tell you what it thinks you want to hear.
So how do you craft a good prompt for getting honest feedback? I’m glad you asked…
2. You know what’s really good at writing AI prompts? AI!
I often tell an AI what I want at a high level and have it craft a detailed prompt. (Which I’ll then review – remember rule 1 – Don’t trust AI!) I’ll then start a new chat, possibly with the same AI system, possibly with another, and use that prompt. It works well.
Mind you, AI-crafted prompts can get long and detailed. For example, here’s what I got when I asked Claude to craft me a prompt that would give me honest feedback on a screenplay:
You are an experienced feature film development executive, script reader, and story analyst with experience across studio, independent, and festival circuits. I am submitting this screenplay to a competition and need brutally honest, professional feedback — not encouragement. Do not soften criticism or bury it in praise. Be direct. Assume I am a serious writer who can handle honest criticism. Do not hedge. Give me your actual assessment.
Except the actual prompt included three hundred more words with details of the dimensions that should be judged. AI is wordy!
Notice how many times the prompt asks for honesty. I count ten ways of it saying, “Don’t butter me up.” That’s what is often required to get the AI to not flatter you.
And believe me, the feedback I got was not flattering. Painful to read, like good feedback often is. But useful.
3. You know what’s good at reviewing the work of AI? AI!
Like I said, always check the AI’s work. But it doesn’t hurt to get other checks on the AI’s work. And AI turns out to be particularly good at checking its own work.
Whenever I use AI to write some computer code, I spin up a second chat and ask it to review the work. And interestingly, it always finds issues, often some bad bugs. Even when the same AI system wrote the code originally.
Which shouldn’t be surprising. After all, when I review my own work, I usually find problems. Why should AI be any different?
But also use different AIs to check the AI’s work. Use Claude to write a document, then have ChatGPT review it. Get Gemini’s opinion too, while you’re at it.
I call these three my AI Braintrust. I spend a lot of time bouncing between them. Because the different AIs will often find different things – it’s like submitting the same document to different human reviewers who all have different strengths and different points of view.
But even though the AI reviewed the results, you need to check it also. And always review the feedback itself before you adopt it. Because we don’t trust AI, right? Always check the AI’s work, even when what you’re checking is AI checking AI’s work.
4. If you have the AI do something long and complex, have it keep notes.
Often a task is large enough that it should be broken down into multiple steps. (AI tends to do better with smaller steps, though you can also have the AI help you prepare the high-level plan that lists the steps.) If you do that, and have the AI work through those steps, have it keep interim notes.
Maybe you’re creating a large complex business plan. Maybe you’re building a large and complex piece of software. Maybe you’re using the AI to help write a full novel.
Those are tasks that will probably require multiple sessions with the AI. At the end of each session, ask the AI to generate a document with notes about the things that were decided in that session. Because that document will be useful in your next session. And without it you may find the AI relitigating old decisions.
And hey, you might find those notes useful too!
5. If you have preferences for how an AI works, write up a preferences document to share with it.
I have strong opinions about some craft matters, both in filmmaking and software development. So I put together a document that includes some of those opinions, my rules of the road. When I work with the AI, I first give it the document.
Maybe you prefer short sentences. Maybe you want to use as many polysyllabic words as possible. Maybe, if you’re a programmer like me, you really don’t like raw numbers in your code and want named constants instead.
Put all that in your preferences document.
(But don’t trust the AI to always follow your rules. Make sure you review its work. Rule 1, right?)
6. Create new chats often. And choose the right model.
This is particularly important if you do a lot of work with AI. Because AI typically rations usage by what they call Tokens. And the longer a chat runs, the more tokens it uses.
If you find yourself running low on your monthly token quota, try starting new chats. Anytime you move to a different topic or task, start a new chat. Consider breaking large tasks into smaller steps, each in its own chat. Because otherwise, you’re carrying along a whole lot of tokens from earlier in the work, which means you’re going to use up your quota sooner.
And be careful which models you use. Each of the big AI companies have different models. Some of them will use up your quota a lot faster than others. Don’t use the Einstein model if the work can be done by Homer Simpson.
In fact, you might want to pay attention to how you use AI and have a chat with your favorite AI system about how often to start a new chat and what models to use. AI is always happy to help, and it can make a huge difference in your usage and costs.
Find these useful? Come back next time for the surprising third act twist!
Mr. Dzikiewicz can be reached at joe@crimesoftheart.com
