Data cleaning headaches

Recently I've been concerned with messy data. Even if the dataset I'm building is comparatively small (dealing with cca 300 new records daily from 6 fairly similar sources), still the manual work I've had to do to normalize it seems inefficient and ugly.

If I had known that I would be building a large dataset, for example, I would've looked into training a small cleaning program to do this for me. As things are, I thought that a few quick-and-dirty methods would get the job done, and now I have to upgrade them every time I add a new data source.

For illustration purposes, let's say that I'm gathering job postings from three different job board APIs.

One of them allows them to select the job location from a pre-made list of city or municipality names. That's wonderful.

The second one allows users to enter the job location as a string (presumably filtering afterward for inaccurate data, but still stranding me with special characters, non-standard formatting, and different names for the same city).

The third has its own convention for naming cities - municipality, various delimiters, then an optional city name.

At the same time, all three allow job postings to be set as country-wide, or to have several potential locations - for example, "Amsterdam, Paris, Rome", or "Amsterdam / Paris / Rome" or even "Amsterdam and Rome or Paris", and altogether it's a fine mess.

If I want to prepare this data for my own schema, I have to account for all these cases and to be able to easily extend the cleaning logic to support new sources in the future.

I'm currently reading Erhard Raim and Hong Hai Do: "Data Cleaning: Problems and Current Approaches", University of Leipzig (2000) and wondering how is it that we have so many frameworks and resources for the same problems in front-end development, for example, while there are comparatively few authoritative sources to rely on for basic data integration and cleaning problems. This answer on StackExchange is intuitive and what I imagine most of us end up utilizing without any guidance. It would just be really nice to have a beginner-friendly set of tools or resource pages for those of us who end up in data science almost by mistake.

GitHub API exercises: location and reputation

Location, location, language

So I've found myself in need of location-specific GitHub users lookup. No problem, right? ?q=location:{location} and off we pop. Not so, friends. While GitHub's search serves based on its own notion of priority, what if I'm looking for Romania-based devs who primarily contribute to Haskell repositories? No, but bear with me: it's 30 minutes of work for someone versed in GH's API. Until Friday, I wasn't one of those. Now I'm 2% closer.

What I'm dealing with so far is this:

1. Make a request to GH for {location}
2. For each result, make a request to GH for {user}
3. Make a request for {user/repos}
4. For each result, grab language
5. Count language
6. Output the biggest result

Straightforward, if I'm not concerned about the user contributing a .css file to a Ruby repo. Let's save that for later.

So of course I could go with this approach, cache those JSONs and filter them at my own chaotic will. Check for changes on a leisurely schedule. Go on my merry way.

Prestige index

What are some relevant metrics of a user's contributions on GH? In an ideal case we're dealing with an open-source contributor whose best works are public. 

• Number of owned repositories [OR]
• Number of stars on ORs
• Number of issues on ORs
• Number of pull requests on ORs
• Number of repositories committed to [CR]
• Number of commits to CRs
• Number of forks merged into CRs
• Number of followers

I'm not including activity into this because I'm not a sociologist. 

What is the relationship? I would say that resolved issues on OR are a solid metric; also, forks of highly-popular repos being merged into the master. I'm certainly missing a ton of things here, but let's say that we're looking for a 0 to 10 score as our final output: how do we put these together?

My initial approach was highly unsatisfactory. At some point, this happened in my calculate-score-per-repo function:

crunchIt.prototype.getScore = function() {
    return this.ns*(this.nf/this.nw);
}

ns = number of stars
nf = number of forks
nw = number of watchers

Naturally I got mathslapped and cowered in a corner, shamed. Who cares about the ratio of forks and watchers? And why should it have the ability to reduce the score of a well-starred repo?

I was wrong and I am ashamed.

So, now I am looking for an actually reasonable way to do this. 

Disclaimer: the 'prestige index' is an exercise meant to give some sort of reasonable result for active GH contributors. Even when it's done it will not be a value judgment. 

Where is Socket.io emitting? A helpful list

From StackOveflow user LearnRPG:

// send to current request socket client
 socket.emit('message', "this is a test");

 // sending to all clients, include sender
 io.sockets.emit('message', "this is a test");

 // sending to all clients except sender
 socket.broadcast.emit('message', "this is a test");

 // sending to all clients in 'game' room(channel) except sender
 socket.broadcast.to('game').emit('message', 'nice game');

  // sending to all clients in 'game' room(channel), include sender
 io.sockets.in('game').emit('message', 'cool game');

 // sending to individual socketid
 io.sockets.socket(socketid).emit('message', 'for your eyes only');

Data science: summer 2014 focus

When I discuss future career plans with my college peers I say, very vaguely: 'I want to work with data'. They give me strange looks and we go on to discuss Czech beer and potential beer-picking apps for iOS.

My point is, I want to work with data acquired through satellites, I want to design and build distributed systems for genome sequencing, and I wanna make huge, complicated, incompatible datasets suddenly work together to produce clear results.

Regrettably, I'm unlikely to crawl near a decent Hadoop site in the near future, and my courseload for the next semester has nary a mention of a single complex query, so there I am taking University of Washington's Data science course on Coursera. I'm watching the videos on my commute to my awesome mainframe-testing internship, which is made more awesome by the fact that before I got it I would also never have had the opportunity to come near a mainframe system.

The course makes me regret that I did not take it before Machine Learning. I'll be repeating ML from September, and thankfully I still have most of my notes and exercises; hopefully I can now practice ML on a custom dataset (I'm thinking Flickr tags, in the vein of the great Vision Lab at Stanford). The data science course is still in its first week, so I have some time to get familiar with Python syntax before the assignment's due.

I'm just struggling to find a way in which I can get some experience working with biological data before applying for my final BSc project. For several months I will be working on my testing and automation skills, and hopefully implement a small version control system in Node, for fun and glory. Where that leaves me is up for debate as some ridiculously talented people two years my junior are currently cross-referencing Nematoda nerve impulses when inside an active blender.

GoogleForms Drupal Module, day 2

Around 36 hours ago, I was in a room filled with iframe-related anguish. Now, that is the type of anguish I understand, the type of anguish that inspires me to pursue heroic and illogical projects and/or go on bad-coding benders.

Bad-coding benders are borne out of good intentions. You want to produce a full-fledged solution to replace an obviously faulty, yet ubiquitous approach. You underestimate the scope of the project. You don't consider that to produce a solution that could be plugged into an existing production website you need to spend approximately 10% of the time 'making it work' and then 90% making it safe and user friendly. With that  100% behind you, you also need to face an additional and unforeseen hurdle in spending as much time making it pretty.

Onto the problem! The currently extant solution for integrating Google Forms into Drupal has stopped development in 2011, and its last running version runs on Drupal 6.x. Not good. It is also entirely huge, making use of the SimpleHtmlDOM library for a task that, turns out, might well be handled by cca 12 lines of beginner's PHP. 

Q. Why is this good?

A. Allows me to play around with how PHP handles DOM. I have never done this before and just this one little step toward scraping a form when given, say, a Google Form [GF] URL or key without the use of iframes puts me far closer to giving it all the functionality the clients/victims care about - custom stylesheets, the entire shizz being wrapped in a Drupal 7.x compatible module, out-the-box jQuery validation and other prettiness.

To approach this part, I've built a three-step process.

Step 1: Simple form page that allows me to paste a Google Form key and click 'Submit' without a care in the world.

Step 2: A flamboyant PHP script majestically entitled 'parser.php' which does several things in an ugly functional manner. It fetches the $_POST values from step 1, strips them of all ugly and dangerous tags and intentions; it goes on to construct a Google Form URL and scrapes the <form> element therefrom; without any additional hassle (for now) it tries to save the scraped form as a unique .html file in a 'forms' directory; failing that, it passes the form data to step 3 in a $_SESSION variable. Clunky, but workable.

Step 3: Our glorious form is presented here with a fresh stylesheet ripe for dynamic loading. 

II. Luckily for my ambitions, my victims' gripes with Google Forms and Drupal don't stop at having to paste the form's iframe in order to get anything done. As many (most) people who love the Google Drive functionality but resent its unfriendliness toward customization (I am not a fan of Google translating everything into the local language, myself), these customers have had to paste the damned form element into their own page, get rid of any and all unwanted elements (hello to the unnecessary and confusing markup!) and style that mess inline.

Q. Why is this good?

A. It allows me to get creative with: stripping unwanted markup from the DOM and making it both accessible and functional; making a custom confirmation page a matter of seconds; ditto for adding a custom stylesheet.

III. The fun doesn't stop there. The true task of this mission is to be able to create a new form straight out of Drupal as a non-technical user. Now, I could more-or-less copy the solution of the previous Google Forms For Drupal Module, or re-code the Ruby solution in PHP, slap it together and pray it works. What this is really good for, however...

Q. Why is this good?

A. If we're already integrating this in Drupal - why stop at painstakingly copying the god-awful creation process of the Gforms? This offers an opportunity for a fun drag-and-drop module [with severe constraints and without using or emulating Masonry] to finally force me into doing something semi-meaningful with JS. It will also mark my first time using AppScript for a somewhat good purpose.

This part requires a full-on Google Form integration, which I'm handling through JSON like a good pupil of the Internet. It also requires building a custom d'n'd solution. 

IV. Validation is, I've learned, an issue of some worry to my new unwitting customers. They've been using a validation script which is, from what I've surmised, simultaneously good and tragically outdated. I am reluctant to butcher an honest developer's licensed code, for obvious reasons; yet I'm even more reluctant to reinvent the wheel. I will need to find a good workable solution that's not almost heavier than the damn CMS they're running on.

Q. Why is this good?

A. It's not, but it does force me to find a decent lightweight validation solution which may well be added to my in-a-bind-toolkit.

V. And finally, how's this integrated into Drupal? Here I'm not talking about making it a module - that seems feasible enough, and will be done over afternoon beer - but I do need to make them a convincing enough UI that just *works*. There will need to be options for anything and everything, from saving and previewing forms [making this a custom post type in addition to a giant pain in the ass] to having a reasonably easy multipage solution. All of this is none the less important than the Huge Happy Task of making myself a Google Forms API from wishes and leprechaun tears. 

Q. Why is this good?

A. I need to start making modules that are actually useful to people, and not just an adorably ugly learning opportunity on a Stark-grey background. Having a complete publishable module, ie. building stuff that is extendable, valid, accessible and has decent UX will be life-changing. 

15-hour Study Week

Apart from my obligations at the actual college I attend, I commit myself to the online courses in this approximate outline:

Machine Learning - 6 hours a week (2 hour lecture-watching while taking notes, and up to 4 hours trying to figure out the programming exercises)

Biostatistics Bootcamp - 4 hours a week (2 hour lecture-watching, 2 hours reading up on the subject and studying notes)

Mathematics - This is somewhat more confusing, as it wholly depends on the week's subject matter. I am planning to reduce the time I spend furrowing my brow at incomprehensible videos and use the profits to do some exercises by myself. For both mathematical courses I am taking online (multivariable calculus and linear algebra) I allot approximately 5 hours a week.

Now add to this the workload for my college, or the classes I am about to begin (namely, Biochemistry), and you can see why I unenrolled from the Stanford Automata class offered on Coursera. The format did not fit my expectations and this was only the second course I signed up for, after ML, so the basically opposite approaches to the workload - videos as long as 40 minutes, for example - disoriented me. I was falling behind and really was not looking forward to the next lesson, so I pledged to study the textbook after my winter study schedule clears up. I really feel that the video format is not well-suited to the subject matter, but this is not to say that I think regular expressions are not important to my studies - quite the opposite; I am simply not in the right mindset to make the best of this class.

Luckily, the new session of Introduction to Mathematical Thinking has been announced, starting on February 4th. You can bet I applied as soon as the notification graced my screen, and was then naturally left reeling as I read the description, wondering as always: 'Is this directly useful? Do I need this?'

Truth is, I may be aiming for quantity over quality here, but I feel that drenching myself in introductory scientific classes is the only way I can get into the right mindset. And if getting the mindset is my goal, then sure as hell IMT is the right class to take. I think it may have even been recommended by my professor at My Real College(tm) on our first class, which was the first time I'd heard a lecturer recommend a MOOC. I had just signed up for Coursera, so this made the lasting impression that I was on the right track.

Bioinformatics via MOOC: introduction and plan

Preamble: I've no substantial experience or knowledge where mathematics, biochemistry, biostatistics or computer science are concerned. I am a student of the HND IT and Computing diploma as governed by Edexcel, and I trust that core curriculum to provide the basics of Computer Science, networks and systems. I will go on to do my Bachelor's in the same program.

My motivation to self-study Bioinformatics is guided by knowing that my chances of getting into a BI master's programme when I finish my Bachelor's are slim to none. Before taking up IT and Computing, I was a biology major with no interest whatsoever in drawing plant morphology. Due to personal reasons I had to end my biology course with nothing to show for it but the finished courses in Chemistry 101, the basics of cell biology and tissue biology, some introductory physics, organic chem, some forgettable biostatistics and the systematics of invertebrates. Not an impressive array of biological knowledge.

As I am studying software development, data manipulation and slightly more involved mathematics, I turn naturally to the crux of information technology and biology: bioinformatics. Via courses offered at MIT Open Courseware, Coursera, Udacity and other individual institutions, using (partly) the 'Bioinformatics SelfStudy Curriculum' from PLOS, I will attempt to fill out the 'systematic gaps in knowledge' and work at getting an education approaching a proper undergraduate degree in Bioinformatics. My goal is to learn; I do not know whether any educational establishment would consider the self-study credentials. I do think they are the best way to learn about the field and not confine myself to the career paths prophecised by the Edexcel diploma records.

My (relevant) courses currently include:
- MIT: Multivariable Calculus
- MIT: Linear Algebra 
- My college: HND Mathematics for software development
- Stanford: Machine Learning (Coursera)
- My college: HND Object-Oriented Programming
- John Hopkins Uni: Mathematical Biostatistics Bootcamp (Coursera)
- Biochemistry (NPTEL)

I start the following courses in January:
- University of Washington: Computational Neuroscience (Coursera)
- Paris School of Engineering: Discrete inference and learning in artificial vision (Coursera)

In February:
- Java programming
- Information systems