RLink is a standard Mathematica package for accessing R functionality from Mathematica. This is a guide on how to set up RLink with arbitrary R installations on various operating systems and various versions of Mathematica.
macOS users with R 3.5 and later, please see this note.
OS X 10.11 El Capitan (and later) users, please see this note.
Motivation
RLink is primarily useful for accessing functionality that is available in R but not Mathematica. Such functionality is usually contained in thirdparty R libraries. R is today the standard computer language for advanced statistics and new computational methods are typically first implemented and published as R libraries. Thus it is essential for RLink to be able to use not only builtin R functions but also arbitrary libraries.
Unfortunately by default RLink uses a builtin installation of R (version 2.14, which is quite old) and—except on Windows—it does not allow adding thirdparty libraries to this installation. For this reason it is most useful to connect RLink to an external installation of R. This guide will show you how to do this.
Setup
Before setting up RLink to use an external R installation, make sure it works correctly with default settings, then quit the Mathematica kernel.
In[1]:= << RLink`
In[2]:= InstallR[]
In[3]:= REvaluate["1+1"]
Out[3]= {2.}
In[4]:= Quit[]
On a computer before where RLink has never been used before, these commands will download some support packages from Wolfram Research servers. This might take a while, let it finish.
Note: On some distributions of Linux, notably recent versions of Ubuntu, the R installation bundled with RLink will not work out of the box: InstallR[]
will fail with “Unable to load dynamic libraries”. This will not prevent external R installations from working, so just continue with the instructions below. For fixing the bundled version of R, see the instructions here.
Now in a fresh kernel, load RLink with Needs["RLink`"]
and proceed according to the instructions for your operating system:

On Windows with Mathematica 9.0.0, 9.0.1 or 10.0.0, RLink does not support R 3.0 or later. R 2.15.3 is the last version that still works.
An external R installation can be used as documented (see under “Details”):
InstallR["RHomeLocation" > "C:\\Program Files\\R\\R2.15.3"]
Mathematica 10.0.1 does support R 3.x so if you have Mathematica 10 please use the 10.0.1 update and the latest version of R.

On OS X, using external R installations is not officially supported, but it does work in practice with any recent version of R. I tested the following with the official R distribution from rproject.org. Note: Users of OS X 10.11 or later must read this first!
First, we need to make sure that RLink can find the necessary libraries, and set the
DYLD_LIBRARY_PATH
environment variable, as follows:SetEnvironment["DYLD_LIBRARY_PATH" > "/Library/Frameworks/R.framework/Resources/lib"]
This needs to be done every time before running
InstallR
and specifying an external installation.If you installed the official R distribution then the path above is correct. If you installed to a nonstandard location or are not using the official distribution, then you’ll need to find where these libraries are and modify the path accordingly.
Then for Mathematica 9.0.0, 9.0.1 or 10.0.0, use
InstallR["RHomeLocation" > "/Library/Frameworks/R.framework/Resources"]
For Mathematica 10.0.1, use
InstallR["RHomeLocation" > "/Library/Frameworks/R.framework/Resources", "RVersion" > 3]
The setting
"RVersion" > 3
is sufficient regardless of which 3.x version you have installed. 
On Linux, the procedure varies by distribution. Generally, simply providing
"RHomeLocation"
will work.For Mathematica 9.0.0, 9.0.1 and 10.0.0, use
InstallR["RHomeLocation" > "/usr/lib/R"]
For Mathematica 10.0.1, use
InstallR["RHomeLocation" > "/usr/lib/R", "RVersion" > 3]
The specific location of R will vary by distribution. On Ubuntu it is generally
/usr/lib/R
, while on Fedora it may be/usr/lib64/R
. On some distributions it may also be necessary to amendLD_LIBRARY_PATH
along the lines ofSetEnvironment["LD_LIBRARY_PATH" > "/usr/lib/R/lib"]
.
RLink should now be ready for use. Test it with
REvaluate["R.version.string"]
to verify that it is using the expected R installation.
Automating these steps
Of course having to jump through all these hoops every time one loads RLink is rather inconvenient. I prefer to put all these steps in a package called RLinkX`
and load that package instead of RLink`
. When using Mathematica 10.0.1, load RLink as
Needs["RLinkX`"]
InstallR[]
When using Mathematica 9.0.0, 9.0.1 or 10.0.0, use
Needs["RLinkX`"]
InstallRX[]
To install this package, download RLinkX.m, and place it in the directory opened by SystemOpen@FileNameJoin[{$UserBaseDirectory, "Applications"}]
. Then open the file and adjust the location of your R installation if necessary.
Credits go to Leonid Shifrin and the Mathematica.SE community for providing many of these solutions.
Related questions on Mathematica.SE:
Note for OS X El Capitan 10.11 (and later) users
Update: R 3.5 and later requires a different workaround. Please try that first.
OS X 10.11 has a System Integrity Protection feature, which is enabled by default. This feature prevents the value of the DYLD_LIBRARY_PATH
environment variable from being passed to child processes. Even if it is set in Mathematica, it will not be passed to the Java executable that RLink uses. Thus the method described above will not work on El Capitan.
So far I have not found any simple and clean workarounds for this problem. The following two approaches should work, but they require some experience using the command line, and modifying the Mathematica installation. Only do this if you are comfortable with the command line and are confident that you can undo the changes. Proceed at your own risk!

Disabling SIP should work, but I have not tested it. Update: I do not recommend disabling SIP, please use the other solution below:

The other method involves patching a library that is part of RLink.

Open Terminal.app and navigate to
SystemFiles/Links/RLink/SystemFiles/Libraries/MacOSXx8664/AllVersions
within Mathematica‘s$InstallationDirectory
. (The location may vary slightly with versions.) There will be a file namedlibjri.jnilib
there. 
Make a backup of this file using
cp libjri.jnilib libjri.jnilib.backup

Rewrite the hardcoded location of the R libraries it uses with the following command:
install_name_tool change @loader_path/libR.dylib /Library/Frameworks/R.framework/Resources/lib/libR.dylib libjri.jnilib
This assumes that R is installed in the standard location within
/Library/Frameworks
. 
Relaunch the Mathematica kernel and proceed as described in the article above. Setting
DYLD_LIBRARY_PATH
can now be skipped.
Note that with this modification, the version of R that ships with RLink will no longer work. It will always be necessary to pass the appropriate
"RHomeLocation"
option toInstallR
from now on. Also, keep in mind that such modifications to Mathematica are unsupported. Please undo these modifications before contacting Wolfram Support with an RLinkrelated issue.I have tested this method with Mathematica 10.3–11.2 on macOS 10.11–10.13 with the latest official R distribution from https://www.rproject.org/. This is the method I am currently using. If a better solution emerges, I will post it here.
Unfortunately, with recent versions of Mathematica and R on macOS, occasionally the R runtime crashes on the first use of
REvaluate
, and the following message is shown: “Crash in lowlevel RLink component or in R runtime. Please reinstall RLink via InstallR”. After usingInstallR[]
a second time, RLink seems to work fine. 
Note for R version 3.5 (and possibly later) on macOS
The above workaround no longer works with R 3.5.0 on macOS (R 3.4.4 is fine). There is an alternative workaround, thanks to Ilian Gachevski, which does not require patching libjri.jnilib
.
To get RLink working with R 3.5 on macOS,

Install the
rJava
package into the external R installation. Start up R, and evaluateinstall.packages("rJava")
. This will download and installrJava
. 
Start up RLink as follows:
Needs["RLink`"] JLink`UninstallJava[]; InstallR[ "RVersion" > 3, "RHomeLocation" > "/Library/Frameworks/R.framework/Resources", "NativeLibLocation" > "/Library/Frameworks/R.framework/Resources/library/rJava/jri" ]

Test RLink by evaluating
REvaluate["R.version.string"]
I have tested this solution with Mathematica 11.3.0 on macOS 10.13.4, R 3.5.0 and rJava 0.99.
Comments !