AB Testing Workshop Service
DataSF offers AB testing design workshops to departments interested in comparing the effectiveness of different forms of communications, e.g. letters or emails.
The redesign of your communication rests on using the principles of Make it Easy, Attractive, Social and Timely or EAST, a framework developed by the Behavioral Insights Team.
Once we design a new letter or communication using the EAST principles, we conduct an experiment (AB test it). The wait it works is that:
- We identify the target population
- Split it randomly into two groups: treatment and control
- Run the text
- Compare the results
If you are interested in a AB Testing Design Workshop, contact Joy Bonaguro for more details.
On this page we will add AB experiments as they are complete.
Treasurer & Tax Collector: A better letter
The Office of the Treasurer & Tax Collector (TTX) collects City taxes on a range of things, including
- Taxes on land and buildings (real property) and
- Taxes on office furniture, boats, equipment, etc (personal property)
The key difference between real and personal property is that the City has the right to seize land and buildings if taxes go unpaid. This gives the City a bit more leverage. But when it comes to personal property, the City relies on self-reported values and cannot simply seize the property if taxes are not paid. This is the difference between a “secured” and “unsecured” tax.
That’s why the Office thought behavioral science insights could help make their tax payment letter more compelling.
Before / After
To improve the letter, the Office partnered with DataSF to conduct a workshop where five teams competed to design a better letter. The winning letter design used the key concepts of making it easy, attractive, social and timely.
The image below provides before and after with the behavioral insights used. In particular, we leveraged concepts of making it easy, using social comparison and norms and making the action items more attractive.
Experiment and Results
To test the new letter, the Office randomly split the 3,495 delinquent taxpayers into 2 groups:
- 1,732 received the old notice (control)
- 1,763 received the new notice (treatment)
Responses were due December 15, 2017. Based on a respose rate of 16.3% for the treatment and 14.0% for the control, we can say with 90% confidence that the new letter improved the response rate. In this case, it improved it by 17%.
TTX is now looking to run experiments on all of its tax letters!
Client Details
Client Team
- Gilbert Jeung, IT, Treasurer & Tax Collector
- Greg Kato, Compliance Director, Treasurer & Tax Collector
- John Krump, Bureau of Delinquent Revenue, Treasurer & Tax Collector
- Lauren Philibosian, Property Tax, Treasurer & Tax Collector
- Margarita Rodriguez, Bureau of Delinquent Revenue, Treasurer & Tax Collector
- Darryl Yee, IT, Treasurer & Tax Collector
Testimonial
Conducting an A/B Test with DataScienceSF was a great experience for the Office of the Treasurer & Tax Collector. They connected us with the Behavioral Insights Team through the What Works Cities initiative. We had training from the state of the industry in using behavioral insights for government work. They expertly facilitated a process where our team quickly learned the major elements of the EAST principles and rapidly put them into practice. We are incorporating A/B testing and behavioral insights into our future communications with taxpayers and look forward to making more evidence-based decisions.
—Greg Kato, Compliance Director, Treasurer & Tax Collector