Home/ Questions/What is the prediction from a rank-deficient fit may be misleading problem, and how to solve it?
Next
Answered
Kaleb Walsh
  • 30
Asked: 2022-05-18T15:28:42+00:00 In:

What is the prediction from a rank-deficient fit may be misleading problem, and how to solve it?

  • 30

. Advertisement .

..3..

. Advertisement .

..4..

I am working on r, but I found the following warning message:

In predict.lm(y[[i]], C2[[i]]) :
prediction from a rank-deficient fit may be misleading

Is there any way to stabilize the issue “prediction from a rank-deficient fit may be misleading”? I read a lot of topics about this, but all of them were trying to install anything. Is this the correct way, or any recommendation for me? Please find the beginning command below:

# Fit regression model to each cluster
 y <- list() 
 length(y) <- k
 vars <- list() 
 length(vars) <- k
 f <- list()
 length(f) <- k
 
 for (i in 1:k) {
  vars[[i]] <- names(corc[[i]][corc[[i]]!= "1"])
  f[[i]] <- as.formula(paste("Death ~", paste(vars[[i]], collapse= "+")))
  y[[i]] <- lm(f[[i]], data=C1[[i]]) #training set
  C1[[i]] <- cbind(C1[[i]], fitted(y[[i]]))
  C2[[i]] <- cbind(C2[[i]], predict(y[[i]], C2[[i]])) #test set
 }
prediction from a rank-deficient
Share

2 Answers

  1. Best Answer
    Expert

    The cause: A matrix that lacks “full rank” is called as the “rank deficient.” The trouble is that predict.lm will issue this warning even if your matrices are full rank (not rank deficient), since predict.lm pulls a quick one under the hood and modifies your full rank input to be rank-deficient by tossing out what it considers worthless data. It then is expressed by giving a warning. Also, it appears that this announcement is a catch-all for other instances, such as when you have too many input features and your data density is too limited, and it is advising you that your predictions are fragile.

    This is an example of predicting while giving full rank matrices, predict.lm is still warning about the lack of rank.

    train <- data.frame(y=c(1,2,3,4),
x1=c(1,1,2,3),
x2=c(3,4,5,2),
x3=c(4,2,6,0),
x4=c(2,1,3,0)
)
test <- data.frame(x1=c(1, 2, 3, 9),
x2=c(3, 5, 1, 15),
x3=c(5, 9, 5, 22),
x4=c(9, 13, 2, 99))
library(Matrix)
cat(rankMatrix(train), "\n") #prints 4, is full rank, good to go
cat(rankMatrix(test), "\n") #prints 4, is full rank, good to go
myformula = as.formula("y ~ x1+x2+x3+x4")
model <- lm(myformula, train)
predict(model, test)
#Warning: prediction from a rank-deficient fit may be misleading

    Solution: Disregard the warning if predict is providing positive results. But if  predict.lm delivers its opinion based on a lack of perspective, you can eliminate errors on the predict step as follows:

    options(warn=-1) #turn off warnings
predict(model, test)
options(warn=1) #turn warnings back on
    • 18

  2. body(predict.lm) allows you to inspect the predict function. This line will appear: 

    if (p < ncol(X) && !(missing(newdata) || is.null(newdata))) 
 warning("prediction from a rank-deficient fit may be misleading")

    This warns you if your data matrix rank is less than the number of parameters that you wish to fit. It can be invoked by using collinear covariates. 

    data <- data.frame(y=c(1,2,3,4), x1=c(1,1,2,3), x2=c(3,4,5,2), x3=c(4,2,6,0), x4=c(2,1,3,0))
data2 <- data.frame(x1=c(3,2,1,3), x2=c(3,2,1,4), x3=c(3,4,5,1), x4=c(0,0,2,3))
fit <- lm(y ~ ., data=data)

predict(fit, data2)
 1 2 3 4 
4.076087 2.826087 1.576087 4.065217 
Warning message:
In predict.lm(fit, data2) :
 prediction from a rank-deficient fit may be misleading

    In data, x3 and 4 have the same direction. The multiple of the other is one. You can check this with length(fit$coefficients) > fit$rank

    A second way is to have more parameters than the variables available: 

    fit2 <- lm(y ~ x1*x2*x3*x4, data=data)
predict(fit2, data2)
Warning message:
In predict.lm(fit2, data2) :
 prediction from a rank-deficient fit may be misleading
    • 12

Leave an answer

You must login to add an answer.

Need An Account,