Steven Hillion Posted October 9, 2019 Share Posted October 9, 2019 Question submitted on behalf of a customer... Facing below error when Notebook is executed, this issue is occurring while converting Pandas DF to Spark DF, I guess this issue is because of data type mismatch between Pandas and Spark, one solution is to match the data type by column by column. But is there an inbuilt feature/function fix this issue Link to comment Share on other sites More sharing options...
Nico Rode Posted October 9, 2019 Share Posted October 9, 2019 Hello! Interesting question.. my guess is that you either didn't initialize the pySpark cluster, or import the dataset using the data tab on the top of the page. You can take a look at this video for more information on how to actually achieve this in Team Studio. The first half of the video talks about importing an excel file, but the second half focuses on associating/importing a dataset to a python notebook, and then converting that pandas dataframe to a pySpark dataframe. Hope that helps! Nico Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now