A/B testing push notifications

Last year, the Service NSW Digital Notifications Team ran an A/B test for push notifications in the Service NSW app. The outcome was a 100% increase in open rates.

About our team

The Digital Notifications team helps NSW Government agencies send effective and secure notifications. This includes emails, text messages, push notifications and MyServiceNSW Account Inbox messages. We’re a team of designers, product managers and engineers.  

We get to work with a wide range of different agencies and customers, such as Revenue NSW, Transport for NSW and the NSW Department of Communities and Justice.  The work is pretty interesting and varied.

The problem and our solution

We were asked to send a push notification for the Active and Creative Kids voucher campaign. The goal was to let parents know they could apply for the new 2023 vouchers.  

There was already messaging approved in other channels such as marketing eDMs. Our team was not sure if using this exact messaging would be as effective for push notification as a channel.  We decided to run an A/B test to find out.   

A/B testing explainer

A/B testing (also known as "experimentation”) is when you compare 2 variations of content to learn which performs better. It's shown to users at random.    

A lot of companies use it. You've probably been a participant of multiple A/B tests without realising.  

The results

We tested title copy only.  

• “New year. New vouchers.” (Variation A)  
• “Vouchers for school kids” (Variation B)  

Variation A aligned to eDM messaging. It had a 4% open rate.  

Variation B was our team’s new copy. It had an 8% open rate.  

This means our copy variant had twice as many people opening the notification.

Last year, I attended a course called “Proving ROI with Content Testing” through the UX Content Collective. We were told to aim for a 4% increase with A/B tests, so the fact we doubled open rates with a 100% increase was incredible. It shows how a few small words can really make a large impact!  

Lessons learnt

… when it comes to push notifications, focus on what will benefit the customer. Or in other words, just tell people what they need to know

While “new year, new vouchers” might sound catchy, our assumption was it was better for push notification copy to be precise and straightforward.    

Another assumption was that referring to “school kids” would be more appealing to parents, carers and guardians. It highlights being able to do or give something positive to the important little people in their lives. This seemed to resonate and be more compelling than catchphrases.  

We learnt when it comes to push notifications, focus on what will benefit the customer. Or in other words, just tell people what they need to know. 

Future projects

Our team uses research and best-practice guidance to inform content decisions. We’ve already taken away learnings from this A/B test and applied them in other areas of work.  

This test also proved how helpful it is to have another pair of “expert” eyes look over your content.  

The Digital Notifications team lives and breathes good content. We’re always looking for opportunities to increase conversion and improve customer experience. If you want some advice on your notification content, contact us. We’d love to work with you. 

Want to work with us?

Contact us to learn more about Service NSW digital capabilities.