Monday, February 10, 2014

Three ways to call C/C++ from R

By Ben Ogorek


I only recently discovered the fundamental connection between the C and R languages. It was during a Bay Area useR Group meeting, where presenter J.J. Allaire shared two points to motivate his talk on Rcpp. The first explained just how much of modern R really is C and C++. For illustration, he used the librestats language composition analysis of Core R, which might not have been so interesting if it did not extend to R's contributed packages. Allaire's second point emphasized R's roots as an interactive mechanism for calling code from compiled languages like C and Fortran (which you can also hear from S inventor John Chambers). While I didn't understand all of the finer points at the time, I never thought of R in the same way again.

Wanting to learn more about this connection between languages, I dug into the C calling paradigms of R. My own lack of experience was mitigated by adopting a tremendously limited scope: the function f(x)= 2x. In the course of its implementation, I received advice from respected voices, each highlighting important points about the alternatives. In this article, I'll share what I learned as concisely as possible. So if you have a passing familiarity with C and an interest in R, grab a cup of coffee, pop open a terminal, and prepare to explore .C, .Call, and Rcpp's sourceCpp.

The big three

The .C function interface

Of R's native functions, the .C interface is the simplest but also the most limited way to call C from R. Inside a running R session, the .C interface allows objects to be directly accessed in an R session's active memory. Thus, to write a compatible C function, all arguments must be pointers. No matter the nature of your function's return value, it too must be handled using pointers. The C function you will write is effectively a subroutine.

Our function f(x)= 2x, implemented as double_me in the file doubler.c, is shown below.
void double_me(int* x) {
  // Doubles the value at the memory location pointed to by x
  *x = *x + *x;
To compile the C code, run the following line at your terminal:
$ R CMD SHLIB doubler.c 
In an R interactive session, run:
.C("double_me", x = as.integer(5))
[1] 10
Notice that the output of .C is a list with names corresponding to the arguments. While the above code is pure C, adding C++ code (instead of C) is made possible by using the extern wrapper.


The .Call interface is the more fully featured and complex cousin of the .C interface. Unlike .C, .Call requires header files that come standard with every R installation. These header files provide access to a new data type, SEXP. The following code, stored in the file, doubler2.c, illustrates its use.
#include <R.h>
#include <Rdefines.h>
SEXP double_me2(SEXP x) {
  // Doubles the value of the first integer element of the SEXP input
  SEXP result;
  PROTECT(result = NEW_INTEGER(1)); // i.e., a scalar quantity
  INTEGER(result)[0] = INTEGER(x)[0] * 2;
  UNPROTECT(1); // Release the one item that was protected
  return result;
Unlike our experience with the .C interface, double_me2 is a function and does return a value. While that appeals to intuition, no matter what the native input and output types, they must now live in a SEXP object. To code double_me2, you must know that there's an integer in the input x, and extract it as if it were the first item in a C array. For the return value, you must add your integer result to a SEXP object in an equally unnatural way. The PROTECT function must be used to prevent R's automatic garbage collection from destroying all the objects.

As before, use R at the command line to compile doubler2.c:
$ R CMD SHLIB doubler2.c 
Back in the R interactive console, the steps are very similar.
.Call("double_me2", as.integer(5))
[1] 10
Notice now that the output is an integer vector instead of a list.

Rcpp and the sourceCpp function

The .C and .Call examples above owe a debt to Jonathan Callahan's entries 8 and 10 of his Using R series. When the examples started working, I tweeted to share my excitement. An hour later, I saw a familiar face:

Let's check it out.

In terms of the code alone, it's easy to see where Hadley is coming from. It's readable, looks just like standard C++ code, and features data types that make intuitive sense. Our simple function is implemented below, saved in the final static file doubler3.cpp (though, in all humility, it's really just C).
#include <Rcpp.h>
using namespace Rcpp;
// [[Rcpp::export]]
int double_me3(int x) {
  // takes a numeric input and doubles it
  return 2 * x;
I'll refer you to Hadley's article High performance functions with Rcpp for details on Rcpp, but for now, note the "// [[Rcpp::export]]" comment, necessary before each C/C++ function, and the updated #include statement. Most importantly, notice how the pointers and SEXP objects have been replaced. Just like our original function f(x), double_me3 takes one integer input and returns one integer output.

After installing the Rcpp package, we're back to the console one final time.
[1] 10
With Rcpp, the function is waiting for us in the global environment, without even compiling at the command line. Pretty convenient!


With the disclaimer that I am a C++ novice, I summarize my thoughts on the three options below.

I like the simplicity of the C code written for the .C interface. It doesn't rely on external header files and it is possible to test using a C compiler alone. On the other hand, I don't like that a function has to be morphed into a subroutine that uses pointers.

In the .Call interface, SEXP objects are also pointers, though that is perhaps superfluous. My biggest complaint is the verboseness that was added to our short example. As Jonathan Callahan points out, .Call "requires much more knowledge of R internals [but] is the recommended, modern approach for serious C [and C++] programmers."

After seeing Rccp in action, it's not hard to understand why Hadley sent me directly to Rcpp. The C code looks great, there were fewer steps, and our function was ready for us inside the R global environment.

Perhaps you're wondering if there is any reason not to use Rcpp. According Murray Stokely, Google Software Engineer, it could be risky on a very large project. "Rcpp's heavy reliance on C macros can make it unsafe to use with large code bases," says Stokely. For example, the Rcpp FAQ (Section 3.9) describes an unresolved issue casting 64-bit integer types. The implication, Stokely explains, is that a loss of precision could occur without any errors or warnings. The Rcpp FAQ considers such examples "corner cases," and perhaps the typical user will not have to worry.

Whatever your decision, I wish you the best on your C++ journey!


Thanks to Max Ghenis and Mindy Greenberg for expedited proofreadings. Keep up with ours and other articles featuring the R language on R-bloggers.

Monday, February 3, 2014

NLSdata: an R package for National Longitudinal Surveys

This article was first published on analyze stuff. It has been contributed to Anything but R-bitrary as the third article in its introductory series.

By Ben Ogorek


Alongside interstate highways, national defense, and social security, your tax dollars are used to collect data. Sometimes it’s high profile and relevant, like the census or NSA’s controversial PRISM surveillance program. Other times it’s just high profile, like the three-billion dollar brain dataset that nobody has figured out how to use. Then there are the lower profile data sets, studied by researchers, available to the public, but with enough barriers to discourage the data hobbyist.

In this article, we’ll work with the NLSY97, a National Longitudinal Survey (NLS) that follows approximately 9,000 youths, documenting "the transition from school to work and into adulthood." The core areas of the survey are education and employment, but subject areas also include relationships with parents, marital and fertility histories, dating, life expectations, and criminal behavior. The NLS data sets are rich, complex, and full of insights on the “making” of a human being.

In this article I introduce NLSdata, an R package facilitating the analysis of National Longitudinal Survey data. I'll discuss the NLS Investigator, a web utility offered by the BLS to download a packet, and demonstrate how the NLSdata package helps to extract value from it. This culminates in the analysis of the belief, "I don't need religion to have good values," and how it has changed with age.

The data

For providing National Longitudinal Survey to the public, the Bureau of Labor Statistics provides an online tool, the NLS Investigator. This article will refer to included files within the Investigator subdirectory of the NLSdata package, but information on how to pull additional data from the NLS Investigator is provided in the Appendix.

While you don’t need to visit the NLS Investigator to run the following examples, the files it provides are the NLSdata package’s raison d’etre and deserve a brief discussion. There are two key files: a structured data file and a semi-structured metadata file called a "codebook."

The data file is in a .csv format and can contain thousands of columns. While it is highly structured, it will probably be unintelligible (see the image below for an example).

The codebook is a text file with a .cdb extension. While it is intelligible, it is only semi-structured (below is an excerpt for one of the training variables).

The NLSdata package

The NLSdata package is currently on GitHub. The easiest way to install it is with the devtools package.
After installation, load in the library and read in the codebook and data files in NLSdata’s Investigator folder.
codebook <- system.file("Investigator", "Religion.cdb", package = "NLSdata")
csv.extract <- system.file("Investigator", "Religion.csv", package = "NLSdata")
The first key step is to create an "NLSdata" object using the CreateNLSdata constructor function. You will then supply the filepath for the codebook and the data. For now, I'll note that there are missing data issues out of the scope of this introduction.
nls.obj <- CreateNLSdata(codebook, csv.extract)
[1] "NLSdata"
[1] "metadata" "data" 
The element "data" is a data frame with variable names that correspond to those in the codebook, but with the survey year appended. Factor labels are added for categorical factors and logical factors have been properly converted. The unique key will always be respondent identifier PUBID.1997.
head(nls.obj$data[order(nls.obj$data$PUBID.1997), c(2, 8, 9, 11)])
     PUBID.1997 YSAQ_282A2.2002 YSAQ_282A2.2005 YSAQ_282A2.2008
5203          1           FALSE            TRUE            TRUE
1918          2            TRUE            TRUE           FALSE
6687          3            TRUE              NA              NA
3682          4            TRUE            TRUE            TRUE
1730          5            TRUE           FALSE           FALSE
2838          6            TRUE           FALSE           FALSE
The metadata element is a list that contains information about each variable in the data set, including the original codebook chunk.
[1] "YSAQ_282A2.2005"


[1] "2005"

[1] "S6316800"

 [1] "S63168.00    [YSAQ-282A2]                              Survey Year: 2005"
 [7] "I don't need religion to have good values."
 [9] "UNIVERSE: All"
(The element "chunk" has been modified for presentation.)

Attitudes about religion through time

The YSAQ_282A2 values indicate each respondent's answer to the question, "I don't need religion to have good values," but the wide format makes it awkward. Thus the NLSdata package provides the function CreateTimeSeriesDf to coerce a portion of the data frame into a long format. Under the hood, it’s using the reshape2's melt function.
religion.df <- CreateTimeSeriesDf(nls.obj, "YSAQ_282A2")
head(religion.df[order(religion.df$PUBID.1997), ])
      PUBID.1997 YSAQ_282A2 year
5203           1      FALSE 2002
14187          1       TRUE 2005
23171          1         NA 2006
32155          1       TRUE 2008
1918           2       TRUE 2002
10902          2       TRUE 2005
Looking at the cell counts, we can see something is strange with the year 2006.
(cell.counts <- with(religion.df, table(YSAQ_282A2, year)))
YSAQ_282A2 2002 2005 2006 2008
     FALSE 4014 3306    0 3213
     TRUE  3829 3679    2 3970
Since only two people answered the question that year, I’m choosing to exclude it from the analysis.
religion.df <- religion.df[religion.df$year != 2006, ]
For the purpose of this article, I want both simplicity and protection from obvious confounding. Thus I'll enforce balance over respondents and years via the ThrowAwayDataForBalance function, which keeps records for respondents who answered a given question in every observed year.
religion.df <- ThrowAwayDataForBalance(religion.df, "YSAQ_282A2")
2002 2005 2008 
6013 6013 6013 
1 2 4 5 6 9 
3 3 3 3 3 3
After deletions, the final cell counts are shown below:
(cell.counts <- with(religion.df, table(YSAQ_282A2, year)))
YSAQ_282A2 2002 2005 2008
     FALSE 3077 2882 2682
     TRUE  2936 3131 3331
We can test whether there is evidence of a changing response distribution by a Chi Square test for association.
        Pearson's Chi-squared test

data:  cell.counts
X-squared = 51.9902, df = 2, p-value = 5.134e-12
There is strong evidence that the likelihood of answering "yes" is not constant over time. Recall that these are the same people in all three years.

And here are the actual proportions.
(proportions <- aggregate(religion.df$YSAQ_282A2, 
                          by  = list(year = religion.df$year),
                          FUN = mean, na.rm = TRUE))
  year         x
1 2002 0.4882754
2 2005 0.5207051
3 2008 0.5539664
In the plot below, I chose a range of 0.40 to 0.65 for the agreement proportion. I believe this is a range which, if covered, would represent a meaningful shift in attitudes.
plot(x ~ year, data = proportions, ylim = c(0.4, 0.65), type = "b",
     ylab = "Proportion agreeing with statement",
     main = 'Belief: "I don\'t need religion to have good values"')


The NLSdata package is still very much a work in progress, and I fully expect that certain untested variables available from the NLS Investigator will cause problems. Reports of such occurrences and general feedback is encouraged and appreciated. The NLS data set itself has many interesting variables. With minimal effort, we determined that beliefs about religion and values changed through time. Many more interesting relationships, those involving education, training, employment, relationships, and even religion, wait to be uncovered. I hope that NLSdata makes the search for these relationships easier and more accessible.


  • Thanks to Max Ghenis, for convincing me to analyze real data and not the output of rnorm(100). He also provided multiple suggestions that improved this article.
  • Thanks to Mindy Greenberg, who deserves a title like "Editor in Chief" for her many contributions to style and readability.

Appendix: using the NLS Investigator

Navigate your browser to the NLS Investigator homepage and sign up for an account. Once you've logged in, you'll be able to choose a survey. In this article, I'm working with NLSY97.
  • the required PubID identifier, which serves as a primary key
  • demographic variables (gender, ethnicity)
  • survey metadata (e.g., release version)
There are literally thousands more under the "Variable Search" tab.

The many variables is due to a massive flattening of the data. The primary ways in which this flattening occurs is summarized below:
  • Each survey round (year) gets a new variable name even with the same question text.
  • Questions are tweaked for different subpopulations, called “universes,” and are repeated.
  • Nested lists called “rosters” add multiple columns for collections like jobs (job1, job2, ..., job 9).
The Investigator’s documentation explicitly warns, “Do not presume that two variables with the same or similar titles necessarily have the same (1) universe of respondents or (2) coding categories or (3) time reference period.” After selecting variables, click the tabs “Save / Download” and then “Advanced Download.” Make sure to select the codebook and the comma-delimited data file.