Recently I've been concerned with messy data. Even if the dataset I'm building is comparatively small (dealing with cca 300 new records daily from 6 fairly similar sources), still the manual work I've had to do to normalize it seems inefficient and ugly.
If I had known that I would be building a large dataset, for example, I would've looked into training a small cleaning program to do this for me. As things are, I thought that a few quick-and-dirty methods would get the job done, and now I have to upgrade them every time I add a new data source.
For illustration purposes, let's say that I'm gathering job postings from three different job board APIs.
One of them allows them to select the job location from a pre-made list of city or municipality names. That's wonderful.
The second one allows users to enter the job location as a string (presumably filtering afterward for inaccurate data, but still stranding me with special characters, non-standard formatting, and different names for the same city).
The third has its own convention for naming cities - municipality, various delimiters, then an optional city name.
At the same time, all three allow job postings to be set as country-wide, or to have several potential locations - for example, "Amsterdam, Paris, Rome", or "Amsterdam / Paris / Rome" or even "Amsterdam and Rome or Paris", and altogether it's a fine mess.
If I want to prepare this data for my own schema, I have to account for all these cases and to be able to easily extend the cleaning logic to support new sources in the future.
I'm currently reading Erhard Raim and Hong Hai Do: "Data Cleaning: Problems and Current Approaches", University of Leipzig (2000) and wondering how is it that we have so many frameworks and resources for the same problems in front-end development, for example, while there are comparatively few authoritative sources to rely on for basic data integration and cleaning problems. This answer on StackExchange is intuitive and what I imagine most of us end up utilizing without any guidance. It would just be really nice to have a beginner-friendly set of tools or resource pages for those of us who end up in data science almost by mistake.
No comments:
Post a Comment