Experiment: Does AI Write Better X (Twitter) Posts Than Humans?

Posted on August 11, 2023 by Insights Desk

At this point, you’ve heard it all about AI. AI can help you write better. AI can help you create faster. But can it help your social captions perform better?

As a busy social media marketer, anything that promises to make the content creation process easier and faster is appealing. And with tools like ChatGPT making AI accessible and user-friendly, it’s worth checking out for writing captions.

Using AI to write social posts, specifically on Twitter, also begs the question: Do AI-written captions impact engagement and reach? Does the Twitter algorithm penalize tweets written by AI? Can it really even tell if a Tweet is written by AI? I ran an experiment to find out.

Craft perfect posts in seconds

OwlyWriter AI instantly generates captions and content ideas for every social media network. It’s seriously easy.

Start free 30-day trial

Hypothesis: Human-written X (Twitter) captions will get more engagement and reach than AI-written captions

My hypothesis for this experiment is that human-written tweets would get more engagement and reach than AI-written ones.

Why? Well, I’ve been on X (formerly Twitter) for more than ten years, so by this point, I should know my own audience best. Plus, humans have been writing for thousands of years and AI has only been doing this for 2 — that has to count for something, right?

But before I conducted the experiment, I wanted to get more insight from someone who works directly in social every day. So I asked one of Hootsuite’s social experts, Trish Riswick, about her experience with AI and if she’s noticed any difference in engagement compared to human-written captions.

Riswick shares that the Hootsuite social team has experimented with using AI to write social captions in a couple of ways.

“At Hootsuite, we have two different types of tweets that we put out,” says Riswick. “We put out fun, linkless, personable tweets for brand awareness. And we also put out stuff that is purpose-driven for when we want people to take an action, like when we have a product update or a webinar that we want people to sign up for.”

Riswick says that she has noticed a difference in engagement and reach when she uses AI for purpose-driven content.

“We see really great engagement and reach when we’re using AI for bottom-of-the-funnel posts because it’s able to deliver the information quicker than we would be able to if we were writing it,” says Riswick. “When we give AI a specific, straightforward job (that doesn’t rely on brand humor or knowledge of current events), the results help us create effective CTAs or a different way of relaying that message.”

On the flip side, she has not noticed an improvement in engagement or reach when using AI to create brand awareness content.

“Those are the things that AI tools don’t really capture as well as a human does,” says Riswick. “Being able to speak in your brand voice is something AI is still learning. AI tools provide a great starting point and help speed up the process, but human oversight and editing is still a must.”

Methodology

To test whether human-written Twitter captions generate more engagement and reach than AI-written ones, I came up with three sets of tweets to share on my personal Twitter account.

I started by sharing my own tweet in my usual tone and style. Then, I prompted ChatGPT to come up with a similar tweet so the two could be fairly compared.

For example, one of my personal tweets was a tip for fellow freelancers. I then gave ChatGPT a simple prompt to replicate my tweet:

Source

While there are a ton of creative prompts you can give ChatGPT, I gave the AI tool some prompts that were on the simpler side, like the one above.

I wanted to keep it pretty bare-bones to give ChatGPT the freedom to generate an organic result rather than something that was heavily refined. (The idea is to see if we can get the same results while saving time, after all.)

For this experiment, I posted four tweets that were text only and two tweets that included a link. Let’s dive into the results!

Results

I checked the performance of each tweet about 24 hours after I posted using Hootsuite Analytics. Here are the results:

	Human captions		AI captions
Post	Eng. rate	Reach	Eng. rate	Reach
Post with link	8.71%	781 impressions	11.43%	35 impressions
Text post 1	2.2%	227 impressions	8.26%	242 impressions
Text post 2	7.22%	97 impressions	1.92%	52 impressions

At first glance, it’s clear that the AI-written tweets had a higher engagement rate than my tweets two out of the three times. However, two out of the three AI-written tweets also had fewer impressions than the human-written ones.

What do these results mean? Let’s break down each of these tweets in more detail.

Tweet #1: Tweet with a link

As a writer, I often turn to Twitter to share my recent work. So my first tweet for this experiment was a tweet with a link promoting an article I wrote.

In the tweet, I summed up the article and provided a little insight into why someone may want to read it by leading off with a personal fact. This is a pretty standard formula I use for my tweets.

Source

Here are the results of this tweet, according to Hootsuite Analytics. The tweet generated an 8.71% engagement rate, 68 total engagements, and 781 impressions.

As I said, it’s normal for me to share my work on Twitter. I also like to add some personal background to these types of tweets to provide context so people are more compelled to read them.

On top of all that, the brand I mentioned in the tweet (@Wethos) retweeted it, which undoubtedly helped boost the number of impressions.

I asked ChatGPT to create something similar using another article I wrote for the same brand, so there was as little bias as possible. Here’s the tweet ChatGPT came up with:

Source

And here are the results of the AI-generated link tweet. This tweet received an 11.43% engagement rate, 4 total engagements, and 35 impressions.

While this tweet technically generated a higher engagement rate, the number of impressions wasn’t even close to the number the human-written tweet received. I’d also note that in my opinion, it’s less personal-sounding than the tweet I wrote.

And unlike my first tweet, the account I mentioned didn’t interact with this post. To be fair, I shared the two tweets that mentioned the brand about a week apart. There’s a chance that they were less likely to reshare again in such a short time frame.

There’s no doubt that the retweet on the first link tweet boosted the number of impressions, and not getting extra coverage on this tweet resulted in very low visibility.

Tweet #2: Text-only tweet

Another topic I often post about is freelancing. I regularly share tips that I think other freelancers would find useful. The post below is an example of a tweet I’d write on a normal day.

Source

This tweet generated an engagement rate of 2.2%, got 5 total engagements, and 227 impressions.

On paper, these results aren’t much to write home about. But they’re on par with my usual performance, especially considering how small my audience is (fewer than 300 followers). This tweet also performed well above the average Twitter engagement rate of 0.05%.

I asked ChatGPT to recreate a similar tweet by prompting it to write a tweet that shares a tip for freelancers. Here’s what it came up with:

Source

If I’m being honest, I didn’t expect this tweet to perform as well as it did. To me, it sounded a little cheesy and didn’t feel like something I’d say naturally. But the results tell a different story.

According to Hootsuite Analytics, this tweet generated an engagement rate of 8.26% and received 20 engagements and 242 impressions.

One of the main reasons I think this tweet performed better is because it included a hashtag and some emojis. I didn’t include either of those in my initial tweet because I usually don’t.

Despite my hesitation, though, I guess there’s a reason the Hootsuite Blog recommends adding hashtags and emojis to your Tweets as a general best practice after all! That engagement spike is hard to argue with.

Tweet #3: Text-only tweet

My final tweet for this experiment is about what it’s like working for yourself and being your own boss. Here’s the tweet I wrote:

Source

After checking Hootsuite Analytics, I can see that this tweet generated a 7.22% engagement rate, 7 engagements, and 97 impressions

As I learned throughout this experiment, ChatGPT *loves* adding emojis and hashtags when you ask it to write a tweet. So for this prompt, I specifically asked it *not* to include any emojis or hashtags.

Though the hashtags clearly made a difference last time, I wanted to see if the AI could generate content that would be more similar to the tone and style of my usual content.

Here’s the prompt:

And here’s the tweet ChatGPT came up with:

Source

Here were the results of this tweet. This tweet generated a 1.92% engagement rate, 1 engagement, and 52 impressions.

This tweet is text only, so it’s very similar to the human-written one. But it generated a much lower engagement rate and received fewer impressions.

Why were engagement and reach so low this time? Well, as far as tone goes, while the AI-generated content is similar to my style of tweets, it sounds pretty generic.

My initial tweet was a little more playful, but ChatGPT’s version is pretty flat — and the numbers back that up. Only one person interacted with this tweet, which goes to show how unnoticed it went in the feed.

What do the results mean?

Overall, the AI-written posts performed better than human-written tweets — at least in terms of engagement.

If you’re a busy social media manager and your primary success metrics are reach and engagement, these results prove that AI can help you create content that follows general best practices and will likely perform well.

Still, even though some of the tweets written by ChatGPT had a higher engagement rate than the ones I wrote, the interactions weren’t necessarily as meaningful.

For example, the AI-written link tweet generated a higher engagement rate than the human-written one, but it had very few interactions. The one I wrote received 16 profile visits, 15 link clicks, and even a few bookmarks, according to Twitter’s native analytics. In my experience, those interactions are more likely to lead to work, so I consider them more valuable.

Top takeaways

Here are more of my insights from this experiment:

These tools can be a major time-saver

If there’s one takeaway I have from this experiment, it’s that AI tools are fast. These tools can crank out tons of tweets in a matter of seconds.

It can take me a while to write my tweets, and I only share a handful on any given week. So I can only imagine how long it takes to create a week or even a month’s worth of captions for a brand account that’s active on Twitter.

AI tools can be helpful to use when you need to write a big batch of captions but don’t have the time to spend writing from scratch.

Hashtags and emojis help

Another surprising takeaway from my experiment? ChatGPT is well-versed in Twitter best practices. It understands how to write something short and attention-grabbing, and even knows that including hashtags can help with reach and visibility.

To be honest, I had never put much thought into hashtags or emojis before. As a professional writer, I always prioritize the tone and construction of a Tweet more than anything else, and it’s worked well for me (so far).

But after seeing how well ChatGPT’s Tweet with hashtags performed, I’m open to adding a hashtag or two in my Tweets going forward — especially when I want to reach a wider audience.

AI makes brainstorming a breeze

Even if the captions AI tools create aren’t totally in line with your brand’s voice or style, they offer a good starting point and can help you brainstorm ideas. They are also great at repurposing and summarizing long-form content.

While ChatGPT requires a bit of finesse to create social captions, an AI tool built for social media like OwlyWriter AI can make the process a lot smoother.

For instance, OwlyWriter’s “Start from scratch” option allows you to select your desired tone before generating content. This definitely could’ve been useful in creating tweets that align with my writing style.

Try OwlyWriter AI for free. You can cancel anytime.

AI-written captions still need a human touch

We’ve all seen tweets that are obviously AI-written. They usually sound generic, lack context, or just don’t make sense.

If you use a tool to write your tweets or captions, they probably won’t be publish-ready right away. You’ll usually need to review for accuracy and tweak until your posts sound like you.

Ultimately, if you’re going to use AI to write captions (and we really think you should!), think of the output as more of a starting point. You’ll have to use your social expertise to refine your results, but you can probably get them done much faster than if you were starting from scratch.

Start building your social media presence using Hootsuite. Schedule and publish posts directly to X, engage your audience, measure performance, and run all your other social media profiles — all from one simple dashboard. Try it free today.

Get Started

Save time and grow faster with OwlyWriter AI, the tool that instantly generates social media captions and content ideas.

Free 30-Day Trial

The post Experiment: Does AI Write Better X (Twitter) Posts Than Humans? appeared first on Social Media Marketing & Management Dashboard.