Please this one time, I beg your indulgence.

I am not normally a toot-yr-own-horn kind of guy.

But, this:

So a thing is happening that has really got me thinking about my analysis and coding and other stuff.

So I’m taking the opportunity to pick up my game, and have started diving back into Project Euler. In specific, I’m going through my 23 completed problems that were done in Python and Perl, and reworking them in R.

Project Euler, generally speaking, is a site where you can work math problems computationally, submit the answers, discuss them on a forum, and badge up as you go. It really seems like a “Many are called, few are chosen” type of thing.  From their site:

• There are a total of 298314 registered members who have solved at least one problem.
• 4580318 correct solutions have been submitted which is an average of 15.4 per member.
• So far 49766 members have solved 25 or more problems, which represents 16.68% of all members.
• There are 123 outstandingly talented members at the current maximum level (solved 375+ problems).

Which means that I’ve already done more than the median and I’m nearing that 16.68% group. That’s pretty cool. Especially that I started this knowing pretty much jack squat about applied math (doing Euler actually got me inspired to start a BS degree in Applied Math at my local state university, that’s how cool this stuff is) or Python. I knew some Perl but I certainly wasn’t and am still not in any way a guru, ninja, rockstar or bro. I don’t even lift. 

So back to the R part of this. I can hear you complaining already. R has a weird syntax, it’s difficult, etc. All of which are true. But in its own way, it’s kind of an elegant creature. As I go back through these problems, I find myself giggling in that, “Oh, that is beautiful” way that happens when I have a geek win.  I didn’t do that so much with Python or Perl solutions to the Euler problems.

I think a lot of that comes from that when I find myself struggling (read: failing repeatedly) with these in R, it’s almost always because I’m not doing them R-ly.  I’m trying to force the old Python or Perl code into R. But the beauty of R is that it’s a vector language and once you can get your code vectorized, it’s smooth like a buttered bobsled run.

For bonus points, after having finally upgraded This Old Mac (srsly, it’s a 2007 aluminum core duo iMac.  But I’m bound and determined to get another 2 yrs out of it before I sell it and upgrade. Because I’m stubborn that way.) to Mountain Leon from Snow Leonard, I can finally use the GitHub client again. So I decided to learn how to use git and GitHub by pushing my Euler R code out there for the world to see (and laugh at).

https://github.com/ClintWeathers/ProjectEulerInR

So there you have it.  Comment, criticize, fork at will.

Notes for myself of R stuff. Maybe someone will get some use from them.

So I bumped into this problem this week:

I was doing some datamungery of json that I’d gotten out of our Solr engine. Now keep in mind that Solr will return json, xml or csv. But csv is just ugly unless you’re needing stuff to drop directly into a dataframe or Excel and XML wouldn’t work because what I needed was actually being listed as an attribute rather than a result.

<data “The data that Clint needs here”> ,8 </data>

Like that, see?  And using the XML package in R and xmlTreeParse wasnt getting me anywhere because the only thing I could search on was the results. “,8″ in this case.

Eventually I hit on something from StackOverflow that used something like this:

lines2bParsed <- readLines(textConnection(getURL(“http://oursolrserver.company.com?querygoeshere”)  which strings together the Curl request piped to a text connection piped to a readLines.

So text connections in R are freaking cool. They let you read in data from a character vector like it was a file. How awesome is that? Then you can just readLines it and you get actual text that you can do something with instead of some weird XML/json/HTML tree thing object. (Note, I tried readLines(“URL of our solr server with the paramaterized query here”) and it didn’t like it, even a little. It really needed the Curl request for the authentication.)

I digress.

So after I got all of that, I had to munge and clean it.  Basically get it from json to clean data of the type I needed in the format that my next step needed.

A few regexes later and I’ve got it cleaned (It might be totally uncool, but I really like to do my regexes in increments in case something breaks, that way I can isolate the fault), but I only need the lines from the json string that have the stuff I need. Luckily, the stuff I need kind of stands out, so again from StackOverflow I find out how to subset based on some sort of parameter:

cleanedData <- lines2bParsed[grep(“your regex goes here”, lines2bParsed)

But there was a problem. Something in my regex was making the whole thing puke. On an off chance, I tried something I didn’t think would work at all, but did and that’s the whole point of this post: You can use an object in R as your regex in a grep. And probably in lots of other stuff where it calls for a regex.  Check this out:

horribleComplicatedAwfulHeironymousBoschRegEx <- “your reg ex goes here”

Now when you do your subset, you can just drop the name of your object in place of the actual pattern to be grepped. Separate the regex itself as an object and it becomes that much simpler to deal with. Unhairball that stuff.

Simplify, simplify, simplify.