Having just passed the one year mark working as a data scientist for the City, it seems an appropriate time to reflect on what I’ve learned. Prior to this job, I did data science work in the private sector for several years for three large corporations. My motivation for pursuing a public sector job was impact driven: I wanted to make a direct contribution to the public (and I knew the data would be highly interesting!).
Unsurprisingly, I had made assumptions as to how a public service role differs from a private sector role. And, also unsurprisingly, I’ve found I was wildly offbase on a number of assumptions. This post is not meant to declare a winner between public and private data science work. After all, the role of data scientist differs considerably across organizations, fields and sectors. Instead, this post is about what I’ve noticed in my role to be substantively different from my previous roles in the private sector and also areas where I’ve been surprised.
I’ve divided these observations into two groups: the ‘“No surprise there” and the “Well, there’s a surprise.”
“No surprise there”
Resources: May as well get the easy one out of the way. Resources in all forms are scarce in the City. From new hires and office space to supplies and office parties, it’s all in short supply.
KPI’s: Corporations typically rally around a set of goals to be achieved in a predetermined amount of time. These goals are often numerical and have to do with increasing profit margin, acquiring new customers, etc. Having clearly defined goals provides for easier alignment across an organization. In government, goals are often much more squishy. For example, ‘improving the lives of residents’ is a hard-to-measure goal and can be interpreted many ways. Goals differ considerably across departments and can be difficult to align.
Centralized data stores: As a data scientist, the lack of a centralized data store has been the most difficult part of the transition. Our work is across departments - each of which has its own data storage system (or lack thereof). As a result, each project has its own data extraction and hygiene challenges. On the flip side, the data nerd in me often has fun forging through new data territory.
“Well, that’s a surprise”
Trail blazing: Believe it or not, we are making waves at the City. There is tremendous opportunity to improve government through good use of data. Part of our mission is to demonstrate the value of analytics. This often results in new approaches to solve problems.
Creativity: My job is to unpack clients’ data, use it to answer their business problem and present it in a digestible way. We then contemplate service changes with them based on insights generated from the data. To be successful, you have to be innovative and think creatively.
Diversity: The breadth of projects at the City makes for a diverse portfolio. Our work ranges from finding leads for the Department of Environment to predicting home prices for the Assessor-Recorder. The portfolio of data science projects requires a range of statistical methods and techniques.
In short, I’ve learned a lot over the last year. I more deeply appreciate the challenges the public sector faces and have been inspired by the work City servants do day in and day out. If you’d like to read about the projects we completed in our first DataScienceSF cohort, you can check out our showcase. We are already hard at work with our second cohort and will share our results in the coming months.