Jump to content

How can I convert a Pandas dataframe into a PySpark dataframe

Recommended Posts

Question submitted on behalf of a customer...

Facing below error when Notebook is executed, this issue is occurring while converting Pandas DF to Spark DF, I guess this issue is because of data type mismatch between Pandas and Spark, one solution is to match the data type by column by column. But is there an inbuilt feature/function fix this issue

Link to comment
Share on other sites


Interesting question.. my guess is that you either didn't initialize the pySpark cluster, or import the dataset using the data tab on the top of the page.

You can take a look at this video for more information on how to actually achieve this in Team Studio. The first half of the video talks about importing an excel file, but the second half focuses on associating/importing a dataset to a python notebook, and then converting that pandas dataframe to a pySpark dataframe.



Hope that helps!



Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Create New...