Aggregating Aggregation: Finding Relevant Jobs

The job hunting process is like climbing a mountain. Each time the mountain crests, you assume it is the top only to be disappointed when there is another ascent. C'est la vie. To that end, I'm trying to implement some robotic legs to do my grunt work to the next ascent.

This is a function that takes the argument that specify a search to Indeed.com, read the page, find the job links, and scrapes the third party redirect postings for relevant keywords. The non-function implementation works well but this bugs out quickly.

require(scrapeR) Keywords <- c("quantitative","physics","entry level","recent","college graduate", " r ","python","forecast","forecasting","analyst","analytics","analysis", "model","modeling","predictive","prediction","analyzed","predicted", "forecasted","new graduate","recent graduate", "data analyst","data") htmlCleaner <- function(htmlString){ return(gsub("<.*?>","",htmlString))} jobFinder <- function(searchWords,area,radius,pageToSearch){ for (l in seq(1,pageToSearch)){ searchDescrip <- gsub(" ","+",searchWords) area <- gsub(" ","+",area) indeedPage <- paste("http://www.indeed.com/jobs?q=",searchDescrip, "&l=",area,"&radius=",as.character(radius),"&sort=date", "&start=",as.character(l*10),sep="") #finds the HTML page, reads it in, and finds the third party links #by indeeds html format. Eliminates blank space. Calls lines that #refer to a job posting. print(indeedPage) html <- tolower(readLines(indeedPage))#searches the html lines for urls. cleanHtml <- html[html!=""] lines <- grep("jobposting",cleanHtml) for (i in seq(1,length(lines))){ #Creates a chunk of html that has hyperlink. Finds hyperlink. chunkHTML <- cleanHtml[lines[i]:(lines[i]+8)] linkLine <- grep("href=\\\"(.*?)\\\"",chunkHTML) test <- gregexpr("href=\\\"(.*?)\\\"", chunkHTML[linkLine]) hLink <- regmatches(chunkHTML[linkLine],test)[[1]] hLink <- paste("www.indeed.com",substr(hLink,7,nchar(hLink)-1),sep="") jPage <- scrape(hLink,follow=TRUE,parse=TRUE) jPage <- htmlCleaner(as(jPage[[1]],"character")) wordMatches <- vector(mode="numeric",length=length(Keywords)) names(wordMatches) <- Keywords for (j in seq(1,length(Keywords))) { if (gregexpr(Keywords[j],jPage)[[1]][1]>0) {wordMatches[j] <- length(gregexpr(Keywords[j],jPage)[[1]])} else{wordMatches[j] <- 0} } print(wordMatches) print(hLink) } } }


Example inputs as "entry level analyst"==search terms, "sacramento ca"==geographic area, 100==radius, 1==the number of pages to search. The bug appears on the fifth third party scrape, with values==defs. Debugging the function takes me to the line that scrapes the HTML but I'm still not sure where the error comes up. Obviously a work in progress, but if I can automate looking through 60 pages of Indeed.com a day, I can save myself a nice chunk of time.

> jobFinder("entry level analyst","sacramento ca",100,1) [1] "http://www.indeed.com/jobs?q=entry+level+analyst&l=sacramento+ca&radius=100&sort=date&start=10" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 0 0 0 model modeling predictive prediction 0 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 0 [1] "www.indeed.com/rc/clk?jk=0dc735391ec9e5fd" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 1 2 0 model modeling predictive prediction 3 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 33 [1] "www.indeed.com/rc/clk?jk=6dd34d0431d25416" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 0 0 4 model modeling predictive prediction 2 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 3 [1] "www.indeed.com/rc/clk?jk=f186fe17c3346164" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 2 forecasting analyst analytics analysis 0 0 0 0 model modeling predictive prediction 0 0 0 0 analyzed predicted forecasted new graduate 0 0 1 0 recent graduate data analyst data 0 0 7 [1] "www.indeed.com/rc/clk?jk=cacce23a3c446ed7" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 2 0 0 forecasting analyst analytics analysis 0 4 2 3 model modeling predictive prediction 1 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 15 [1] "www.indeed.com/rc/clk?jk=8382c8ba0875d4f8" quantitative physics entry level recent 0 0 0 0 college graduate r python forecast 0 0 0 0 forecasting analyst analytics analysis 0 0 0 0 model modeling predictive prediction 0 0 0 0 analyzed predicted forecasted new graduate 0 0 0 0 recent graduate data analyst data 0 0 0 [1] "www.indeed.com/rc/clk?jk=b56639458dcd2619" Error in value == defs : 'value' is missing

Predictive Modeling of S.F. Crime: First Backtest

I've incorporated multiple simulations per day to produce distributions for crime volume, turning this

"X crimes will happen tomorrow according to my model" to

"X-Y crimes happen 50% of the time according to my model".

Here's a short backtest using the 150 days beginning on Jan. 1st, 2013 to illustrate what's going on. The percentile lines correspond to the 95th, 80th, 20th and 5th percentiles for each day in the simulation. The points are the realized volumes.

I've recalibrated the means used by the model to the 2003 data and ran the model forward 5 years to see how the model accounts for historical data.
A "good" performance:
1. ~60% of volumes should lie between the 20th and 80th percentile.
2. ~15% of volumes should lie in both the 5th to 20th and 80th to 95th range (~30% combined).
3. 10% should lie above and below the 95th and 5th, respectively.

What happened?
1. 50% of historical volumes were within the model-produced 20th and 60th percentiles. Low.
2. 19% were within the model-produced 5th and 20th percentile.  High.
3. 13% were within the model-produced 80th and 95th percentile. Low.
4.  ~17% were above/below forecast 95th/5th.

What does that mean?
1. As suspected, my model underestimates the frequency of abnormally high/low volume days (ie, fat-tailed events).

2. Judging from the 5th-20th and 80th-95th results, the forecasts produce higher volume days more often than lower volume days, which is inconsistent with past observations.

Now what?
The underlying issue is that my model produces volumes by combining random draws from probability distributions. Going back to Probability 101, generating a 95th percentile event will require multiple high draws from the twelve distributions. The benefit of my approach is that I can tinker with means by category (customize) without much problem and independently. The downside is that the combination of draws decreases the consistency of my forecasts with historical observation.

A way around this could be to initiate one draw which will inform the draws from each other category. Rather than combining a number of independent, individual draws to produce a volume, a  percentile could be chosen, passed to the categorical distributions and inform that choice. That would reduce the problem of combined probabilities and better account for high-volume events.
But I'm not being paid and this model is jumping into overfit territory quickly, so I won't.

Predictive Modeling of S.F. Crime: Adding Monthly Tendencies

I have been developing a model for forecasting crime volume in San Francisco, informed through publicly available historical data going back ten years. Volumes are generated through a sum of choices from normal distributions by crime category, as informed from the past data. By shifting the mean of this distribution, I have accounted for day of the week trends (Friday being a higher crime day than Tuesday) and a long-term linear decrease.

The last tweak is to account for monthly shifts. Here's a plot lacking severely in elegance to give a quick idea as to the magnitudes by year. Each year, I took the mean of the twelve months and differenced the actual number by month from the mean.

The clearest trend is that February and December reside below the mean. To include this numerically, I've taken an average of the percentage away from the mean by year. A quick example will make this more clear.

In '03, 13482 crimes were reported in January. The mean of that year was 12741. 13482-12741 ~ 740. As a percentage of the mean, 740/12741 ~5.8%. Doing this for each January, year by year, and averaging those percentages yields 3.5%. Doing this for each month, the list looks like
January ~ 3.5%        ~ 0.00115
February ~ -6.2%    ~ -0.00222
March ~ 3.6%         ~ 0.001185
April ~ -0.6%          ~ -0.000218
May ~ 1.6%             ~ 0.0000538
June ~ -3.8%            ~ -0.001285
July ~ 0.02%           ~ 0.0000711
August ~ 3.5%          ~ 0.001142
September~ 1.6%     ~ 0.000533
October ~ 5.2%        ~ 0.00167
November ~ -3.0%   ~ -0.001011
December ~ -5.6%   ~ -0.00183

As a sanity check, the sum of these should be close to zero. Taking the full, "unapproximated" values in R yields a number raised to the -17th so ~0. To include these percentages as a modifier to a daily volume forecast, I need to translate these numbers into a daily shift. This percentage does not need to be geometric like the long-term decrease because the mean is effectively being reset at each run. On an annual basis, this monthly tweak should redistribute when the crime volume occurs, but not change the total volume.

The second number trailing the percent is the daily shift that accounts for the monthly shift, by dividing the percentage by the number of days in that month and ignoring leap years.

My model is a simple conditional tweak to the means of normal distributions informed by past crimes. To put this new shift in, the past mean calculation,
Mean = Historical Mean * [ (Geometric Average Decrease)^(Number of Forecast Days) + (Day of Week Shift)]
will include a new component so that
Mean = His Mean * [(GeoAvgDec)^(NumOfDays) + (DoW Mod) + (Month of Year Mod)]

The result will be a redistribution of forecast volume which will stay consistent with annual expectations.

Above is the new method including the monthly modifier and below is the previous forecast using only a day of the week shift and a long-term shift. By inspection, it is seen that this model still does not forecast extreme events but inclusion of this new modifier has spread the distribution out rather than clustering so heavily around the downward linear trend.

The next step is to account for the fat tail events, and keeping in mind the longer term goal of putting forecasts in the context of Monte Carlo simulations.

Predictive Modeling of S.F.P.D. Crime: Falling Short

Turning the crank in Python and R, here's my first long term forecast incorporating the tailoring detailed in the last few posts. I didn't spend much time on the plot, hence the funky axis units. The left of the red line is the actual daily volumes from 2003-2013, while after the red line is the forecast volumes. 

So far, a long-term decreasing trend and a day of the week modifier has been included. By inspection, the long-term decrease seems to be doing what it needs to because there is a clear decreasing trend. What about the day of the week modifier?

Percentile results show that the forecast volumes are lower and more tightly distributed than historical volumes. Forecasts are in italics and historical results are in regular font.

Wednesday

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  202.0   339.0   372.0   377.6   414.0   666.0 

  Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  277.0   326.2   344.0   344.7   362.0   422.0 

Thursday

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  197.0   352.0   379.0   382.9   413.0   576.0 

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

    249     311     331     331     349     406 

Friday

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

    6.0   350.0   388.0   388.8   429.0   552.0 

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  292.0   337.0   353.0   356.0   371.8   427.0 

Saturday

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  149.0   325.0   363.0   361.6   393.0   566.0 

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  272.0   316.0   334.0   335.4   355.0   394.0 

Sunday

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

    2.0   313.0   340.0   341.3   369.0   588.0 

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  237.0   291.8   308.0   309.6   326.2   376.0 

Monday

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  190.0   329.0   362.0   362.6   395.0   522.0 

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  256.0   298.0   317.5   318.5   338.0   395.0 

Tuesday

Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  210.0   347.0   376.0   379.4   409.0   553.0 

 Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 

  274.0   311.8   328.0   333.9   352.0   421.0 

In both cases, Friday is the peak day for historical and forecast crime while Sunday is the minimum. At first glance, the modifier looks like it has successfully shifted the daily volumes to better fit historical patterns. 

Takeaways:

• The forecast volumes hug the mean more closely than historical volumes.
  More technically, switching from a normal distribution to a more fat tailed distribution is probably warranted. A fatter tail in the probability distribution will pull more "extreme" values more often, and so forecast volumes will spread away from the mean. When I incorporate the Monte Carlo aspect, this should lead to more accurate percentile results.
• The mean of the forecast volumes are lower than the historical means.
  I have added a decreasing trend, so volumes would be expected to be lower. Additionally, the historical means include the last ten years of crime and so reflect the higher volumes early on in the sample. Comparing instead with a more recent set like the three year period from 2009-2012. Turning the crank on that, the means increase so something seems amiss as that is not consistent from inspection.

Predictive Modeling of S.F.P.D. Crime: Daily and Long-Term Tailoring

My last post established that a linear model of crime volume from 2003 to 2013 has a slope of -25 (incorrectly labeled 23 in the image from the last post), and that there is variation in crime volume associated with the day of the week. This post details how I intend to include this into the model.

My approach is to shift the means of the normal distributions that I am pulling from by applying a number of modifiers, which will incorporate the specific trends I have mentioned.

As a demonstration, assume that the daily decrease in crime is ~1%, Wednesday has 5% lesscrime than a normal day and Tuesday has 5% more crime than a normal day. Tailoring the model to include these tendencies requires shifting the means of these normal distributions. As a concrete example, let's consider thefts. The mean is ~89 per day.

Through the week, the mean of the distribution selected will be changed like this:

Monday	Tuesday	Wednesday
Mean=89	Mean=89[(1-0.01)^1+(0.05)]	Mean=89[(1-0.01)^2+(-0.05)]

The term inside of the brackets is the percentage modifier. The first term accounts for 1% decrease, compounding daily, hence the geometric term.  The second term accounts for the day of the week. I'll explain the origin of each later.


Long-Term Decrease:
Sequentially taking monthly percentage changes in crime volume from 2003 to 2013 produces a series of 120 "returns".The geometric mean comes to -0.212%. Bringing this to our example of an ~89 mean, day one will have 89 crimes. Day two will have 89*(1+Geometric Mean=-0.00212) = 88.81. Day three will have day two's mean, 88.81*(1+Geometric Mean = -0.00212)=88.62. This can be simplified by 89(1+GeoMean)^(Day), where "Day" is the number of the days from the start. Over a year period, the last number for Day would be 365.
The long-term decrease will be accounted for by a term of (0.997877)^(Day of Forecast).

Day of Week Modifier:
This is a lesson in documenting code. I don't remember how I got to these results, but here's what I have noted. Listed in order from Monday to Sunday, -3.3%, 0.4%, 3.5%, 0.3%, 6.6%, 0.4%, -7.8%. The numbers work, in the sense that they sum pretty close to 0 so that they will shift the distributions without affecting the totals. This is important because historical volume informs my model, and I want to stay true to that while tailoring to make forecasts account for the day of the week.
Edit:
Alright, I went back. Here's how I got those numbers.
Each year, take the number of crimes occurring on each day of the week and find the mean. For example, Monday through Friday have 100 crimes each day and Saturday, Sunday have 200 each. The mean is 128. Daily volume can be represented as a percentage above/below the mean. Saturday, Sunday are 56% above it while Monday through Friday are -21%. Summing these percentages gives zero, meaning that the predicted means overly an annual period should be consistent with historical annual means.
Daily Modifier Shift:
Monday, -0.034
Tuesday, 0.002
Wednesday, 0.033
Thursday, 0.001
Friday, 0.067
Saturday, 0.006
Sunday, -0.074

Predictive Modeling of S.F. Crime: Trends

So far, my model generates crime predictions through statistical applications of historical data. Calling it predictive is a stretch because it assumes a stationary, homogeneous environment for crime. Some examples are that daily forecasts do not account for day of the week volume differences or long term crime trends. I explored that a little in the data today and produced some charts to that end.

Long-Term Trends:

Monthly volume was compiled into a list. The result is the plot below, and a best fit trendline that shows crime over the last 11 years has been approximately decreasing at a rate of 23 crimes per month. Significant variance exists on a monthly basis, but there is definitely a long-term decreasing trend which could be included in a model forecasting many years out.

Day of Week Trends:

Below is a barplot that takes crime volume by day, by year from 2003 to 2013. That is, the first seven bars correspond to the number of crimes on Friday, Monday, Saturday, Sunday, Thursday, Tuesday and Wednesday occurring in 2003.






















More demonstrative is to take the mean number of crimes for the days of the week in each year and calculate the distance from that mean. Let's say Friday has 1000 crimes, Monday-Thursday have 500 crimes, Saturday and Sunday have 700 crimes each. The total # is 4400 crimes in the week. The mean is 628. Put another way, 628 crimes occurring each day of the week would reach that [cough, rounding] same number (7*628 ~ 4400). The barplot below takes the distance by day from the weekly mean for each year. We see that Friday and Wednesday frequently lie above the mean, while Sunday is uniformly below the mean.

Takeaways:
1. A long-term decreasing trend would be a useful inclusion to multi-year forecasts. This trend is not strictly decreasing and deviates significantly on a monthly basis.
2. Daily differences matter. Friday has more crime and Sunday has less crime. Including this to differentiate a Friday forecast from a Sunday forecast can solve the ambiguous and interchangeable format of current forecasts.

Predictive Modeling of S.F. Crime: Introduction

My last few posts have approached S.F. reported crime data from a descriptive perspective. My goal was to familiarize the unknown by employing data descriptively and presenting it in simple and intuitive ways. Methods keeping to this theme include:

• natural frequencies, for example, 1 in 5 reported crimes in the Tenderloin were drug related. 
• visualizations, such as barplots and wordclouds to compare relative magnitudes of natural frequencies and maps of San Francisco with the density of various crimes plotted over it (heatmap, of sorts).
• brief summaries of stand-out traits in each district. The Tenderloin has the highest density of crime by area. Violent and drug-related crime are far more frequent than in other districts, and as a consequence, the arrest rate of 70% is double or triple that of the other districts.

The last few weeks I have switched focus from descriptive analysis to predictive analysis. Instead of summarizing what has happened, I want to use relevant historical data to generate forecasts for reported criminal activity.

I'm keeping it simple to start. I organized the data into daily cuts. From there, the crime/day was calculated for high volume crimes individually (crimes occurring more than 2500 times a year such as theft, robbery, burglary, assault, missing persons, non-criminal, other offenses, drugs, warrants, vehicle thefts) and for low volume crimes combined.

The data set of 223 days allows for a histogram of 223 individual values. For the high volume crimes (mean of more than ten a day), their distribution looks normal for a first run approximation. One such example:

 Approximating each high volume crime and the grouped together low volume crimes with a normal distribution gives a mean and standard deviation. A daily crime volume forecast can be produced by pulling randomly from each distribution, giving a # of each crime/day. Summing these, a daily volume is produced. Repeating this process 365 times yields the following barplot. The red line is the annual mean and the blue lines show a standard deviation on either side of the mean.

This is a first run and has little utility for decision making. There is no accounting for daily, monthly or annual trends, such as higher crime volume on Friday. It does not distribute crime by police district, which would be relevant to staffing. More importantly, each day is a random selection from a normal distribution. Another annual generation would likely not resemble this one in anything but the mean and variance.

Where can this be improved?

The short answer is everywhere. The main inclusion needed is realizing daily forecasts in the context of Monte Carlo simulations. With many daily forecasts, I can replace a one number estimate with percentiles. As an example, my day one crime volume was 339. If I ran my annual simulation 1000 times, I could instead offer an estimate through percentiles such as, "95% of outcomes fell below 400 crimes/day". 

This still does not improve the model, except in improving the clarity of results. To make this model useful, I need to include how these crimes are distributed by hourly blocks and police districts. Further, pulling from a stationary mean and variance make this a time independent model. While that might be the best approximation, I am not well enough informed to make that assumption. Finally, long-term or daily trends can be included on further studying the data.