Sign Up

Sign Up to our social questions and Answers Engine to ask questions, answer people’s questions, and connect with other people.

Have an account? Sign In

Have an account? Sign In Now

Sign In

Login to our social questions & Answers Engine to ask questions answer people’s questions & connect with other people.

Sign Up Here

Forgot Password?

Don't have account, Sign Up Here

Forgot Password

Lost your password? Please enter your email address. You will receive a link and will create a new password via email.

Have an account? Sign In Now

You must login to ask question.(5)

Forgot Password?

Need An Account, Sign Up Here

Please briefly explain why you feel this question should be reported.

Please briefly explain why you feel this answer should be reported.

Please briefly explain why you feel this user should be reported.

Sign InSign Up

ITtutoria

ITtutoria Logo ITtutoria Logo

ITtutoria Navigation

  • Python
  • Java
  • Reactjs
  • JavaScript
  • R
  • PySpark
  • MYSQL
  • Pandas
  • QA
  • C++
Ask A Question

Mobile menu

Close
Ask a Question
  • Home
  • Python
  • Science
  • Java
  • JavaScript
  • Reactjs
  • Nodejs
  • Tools
  • QA
Home/ Questions/How to deal with the failed to initialize nvml driver library version mismatch error?
Next
Answered
jailynchamplin
  • 4
jailynchamplin
Asked: June 15, 20222022-06-15T01:14:36+00:00 2022-06-15T01:14:36+00:00In: Error

How to deal with the failed to initialize nvml driver library version mismatch error?

  • 4

. Advertisement .

..3..

. Advertisement .

..4..

I encounter ”failed to initialize nvml driver library version mismatch” error as the title says when trying to run a GPU workload:

Warning Failed 91s (x4 over 2m12s) kubelet, ip-10-0-129-17.us-west-2.compute.internal Error: failed to create containerd task: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused "process_linux.go:432: running prestart hook 0 caused "error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: initialization error: nvml error: driver/library version mismatch\\\\n""": unknown

How can i fix it so the error goes away?

  • 1 1 Answer
  • 669 Views
  • 0 Followers
  • 0
Answer
Share
  • Facebook
  • Report

1 Answer

  • Voted
  • Oldest
  • Recent
  • Random
  1. Best Answer
    hdtutoria Expert
    2022-06-15T01:22:33+00:00Added an answer on June 15, 2022 at 1:22 am

    The cause: This error occurs when NVIDIA drivers on a node are upgraded but the run-time driver information is not up to date.

    The solution: Drain and reboot the worker

    The simplest way to resolve the problem is to reboot the node. After the upgrade, reboot the node to ensure that the drivers are correctly initialized.

    Before deploying new workloads on a GPU worker node, we recommend draining the node, completing the driver upgrade, and then rebooting the node.

     

    • 3
    • Reply
    • Share
      Share
      • Share on Facebook
      • Share on Twitter
      • Share on LinkedIn
      • Share on WhatsApp
      • Report

Leave an answer
Cancel reply

You must login to add an answer.

Forgot Password?

Need An Account, Sign Up Here

Sidebar

Ask A Question
  • How to Split String by space in C++
  • How To Convert A Pandas DataFrame Column To A List
  • How to Replace Multiple Characters in A String in Python?
  • How To Remove Special Characters From String Python

Explore

  • Home
  • Tutorial

Footer

ITtutoria

ITtutoria

This website is user friendly and will facilitate transferring knowledge. It would be useful for a self-initiated learning process.

@ ITTutoria Co Ltd.

Tutorial

  • Home
  • Python
  • Science
  • Java
  • JavaScript
  • Reactjs
  • Nodejs
  • Tools
  • QA

Legal Stuff

  • About Us
  • Terms of Use
  • Privacy Policy
  • Contact Us

DMCA.com Protection Status

Help

  • Knowledge Base
  • Support

Follow

© 2022 Ittutoria. All Rights Reserved.

Insert/edit link

Enter the destination URL

Or link to existing content

    No search term specified. Showing recent items. Search or use up and down arrow keys to select an item.