The web app works seamlessly across laptops and mobile devices. As always, our goal was to make a product that inspires users to immediately jump into exploring the data. The tool needs to be intuitive. It should just make sense.

Meeting these goals requires more than just a well-designed user-interface (as important as that is!) It also requires reconciling different data sources in a way that allows for accurate comparisons. And, critically, the workflow to produce the data must be easy-to-update. The internet is full of beautiful data dashboards that are useless because the underlying data is out-of-date.

Here is a schematic diagram of how we solved these challenges in North Carolina. In short, our process was as follows:

1. Obtain and merge files containing public school information.

2. Obtain and merge files containing private school information. This included collection of original data.

3. Create consistent fields in the public and private data. Merge the datasets.

4. Populate our School Finder web application with the latest data.

As much as practicable, we simplified these steps until new additions to the data could be added immediately with minimal overhead.

Each state maintains school data in different ways. In North Carolina, comprehensive public school records are collected by the state Department of Public Instruction (DPI) who publishes them in a standardized way each year.

Private school information is gathered by a different state agency. The availability and formatting of that private school data has evolved over recent years, and it is less well-maintained than the public school data. For instance, private schools lack consistent unique IDs. Instead, they are simply identified by name and city–two fields which are entered inconsistently across datasets. A school directory file and a school enrollment file might contain different town and school abbreviations.

After collecting publicly available private school data, we worked with PEFNC to develop and field a private school survey collecting additional information. We then added these survey responses to our final private school dataset.

We took care to hard-code changes and modifications to the original source data. This way any changes were permanently documented. This allowed us to make subsequent changes to the underlying data quickly and accurately.

We first began working with PEFNC during the 2022-23 school year. We invested the time to establish a reproducible workflow beginning with the state’s various raw data sources and ending with our customized School Finder app. With this code pipeline in place, we were able to quickly process and publish the next school year’s data, keeping PEFNC’s data as current as possible.