Data Science Story

AB Testing Service and Results to Date

Learn about our AB testing workshop service and about completed experiments across the City!

AB Testing Workshop Service

DataSF offers AB testing design workshops to departments interested in comparing the effectiveness of different forms of communications, e.g. letters or emails.

The redesign of your communication rests on using the principles of Make it Easy, Attractive, Social and Timely or EAST, a framework developed by the Behavioral Insights Team.

Once we design a new letter or communication using the EAST principles, we conduct an experiment (AB test it). The wait it works is that:

  • We identify the target population
  • Split it randomly into two groups: treatment and control
  • Run the text
  • Compare the results

A diagram showing the bulleted steps in an experiment visually

Above are the steps involved in an AB testing experiment.

If you are interested in a AB Testing Design Workshop, contact Joy Bonaguro for more details.

On this page we will add AB experiments as they are complete.

Treasurer & Tax Collector: A better letter

The Office of the Treasurer & Tax Collector (TTX) collects City taxes on a range of things, including

  • Taxes on land and buildings (real property) and
  • Taxes on office furniture, boats, equipment, etc (personal property)

The key difference between real and personal property is that the City has the right to seize land and buildings if taxes go unpaid. This gives the City a bit more leverage. But when it comes to personal property, the City relies on self-reported values and cannot simply seize the property if taxes are not paid. This is the difference between a “secured” and “unsecured” tax.

That’s why the Office thought behavioral science insights could help make their tax payment letter more compelling.

Image showing unsecured versus secure property types.

It is harder to collect taxes on unsecured property like office furniture, boats, or equipment. Unlike land or structures, i.e. secured property, the City can't seize unsecured property in the event of nonpayment.

Before / After

To improve the letter, the Office partnered with DataSF to conduct a workshop where five teams competed to design a better letter. The winning letter design used the key concepts of making it easy, attractive, social and timely.

The image below provides before and after with the behavioral insights used. In particular, we leveraged concepts of making it easy, using social comparison and norms and making the action items more attractive.

An image showing the original letter and the redesigned letter.

The new letter, on the righthand side includes the EAST principles.

Experiment and Results

To test the new letter, the Office randomly split the 3,495 delinquent taxpayers into 2 groups:

  • 1,732 received the old notice (control)
  • 1,763 received the new notice (treatment)

Responses were due December 15, 2017. Based on a respose rate of 16.3% for the treatment and 14.0% for the control, we can say with 90% confidence that the new letter improved the response rate. In this case, it improved it by 17%.

TTX is now looking to run experiments on all of its tax letters!

A bar chart comparing the response rates for the control and treatment.

The rate of response to the new letter (treatment) was 17% higher than the response rate for the old letter. We are 90% confidant that our new letter improved response rates with p=.0515

Client Details

Client Team

  • Gilbert Jeung, IT, Treasurer & Tax Collector
  • Greg Kato, Compliance Director, Treasurer & Tax Collector
  • John Krump, Bureau of Delinquent Revenue, Treasurer & Tax Collector
  • Lauren Philibosian, Property Tax, Treasurer & Tax Collector
  • Margarita Rodriguez, Bureau of Delinquent Revenue, Treasurer & Tax Collector
  • Darryl Yee, IT, Treasurer & Tax Collector

Testimonial

Conducting an A/B Test with DataScienceSF was a great experience for the Office of the Treasurer & Tax Collector. They connected us with the Behavioral Insights Team through the What Works Cities initiative. We had training from the state of the industry in using behavioral insights for government work. They expertly facilitated a process where our team quickly learned the major elements of the EAST principles and rapidly put them into practice. We are incorporating A/B testing and behavioral insights into our future communications with taxpayers and look forward to making more evidence-based decisions.

—Greg Kato, Compliance Director, Treasurer & Tax Collector