Python College

I am having some troubles reading the following CSV data in UTF-16: FullName, FullLabel, Type TEST.… Read more How To Read Csv File With Additional Comma In Quotes Using Pyspark?

I am using spark 2.1.0. I am not able to create timestamp column in pyspark I am using below code s… Read more Pyspark Creating Timestamp Column

My data frame looks like - no city amount 1 Kenora 56% 2 … Read more Replace Column Values In Spark Dataframe Based On Dictionary Similar To Np.where

I have installed pyspark through pip but unable to open it. It shows following error . Users/sonv… Read more Unable To Open Pyspark In Mac Os

I'm working with Spark 1.3.0 using PySpark and MLlib and I need to save and load my models. I u… Read more What Is The Right Way To Save\load Models In Spark\pyspark

I am not sure is this possible in pyspark. I believe it should be just that i am not winning here :… Read more Pyspark Dataframe - How To Pass String Variable To Df.where() Condition

I built a python module and I want to import it in my pyspark application. My package directory str… Read more Pyspark Import User Defined Module Or .py Files

I am following the H2O example to run target mean encoding in Sparking Water (sparking water 2.4.2 … Read more H2o Target Mean Encoder "frames Are Being Sent In The Same Order" Error

I want to be able to use a Scala function as a UDF in PySpark package com.test object ScalaPySpark… Read more How To Use Scala Udf In Pyspark?

Is there any way to use custom record delimiters while reading a csv file in pyspark. In my file re… Read more Read A File In Pyspark With Custom Column And Record Delmiter

This UDF is written to replace a column's value with a variable. Python 2.7; Spark 2.2.0 import… Read more Pyspark Udf On Withcolumn To Replace Column

I wrote a pyspark script that reads two json files, coGroup them and sends the result to an elastic… Read more Pyspark: Ship Jar Dependency With Spark-submit

I am running Spark programs on a large cluster (for which, I do not have administrative privileges)… Read more Numpy And Static Linking

Below is my spark data frame a b c 1 3 4 2 0 0 4 1 0 2 2 0 My output should be as below a b c 1 3 … Read more Pyspark Dynamic Column Computation

The data is as below. [Row(_c0='ACW00011604 17.1167 -61.7833 10.1 ST JOHNS COOLIDGE FLD … Read more How To Read A Fixed Character Length Format File In Spark

I'm running a Lubuntu 16.04 Machine with gcc installed. I'm not getting gensim to work with… Read more How To Get Cython And Gensim To Work With Pyspark

So what I am trying to do is simply to convert fields: year, month, day, hour, minute (which are of… Read more Unsupportedoperationexception: Cannot Evalute Expression: .. When Adding New Column Withcolumn() And Udf()

My spark version is 2.4.0, it has python2.7 and python 3.7 . The default version is python2.7. Now … Read more How To Correctly Set Python Version In Spark?

I have a dataframe with a very large number of columns (>30000). I'm filling it with 1 and 0… Read more Efficient Column Processing In Pyspark

I'm having a pyspark dataframe with 2 columns. I created a azure cache for redis instance. I wo… Read more Is There A Way To Write Pyspark Dataframe To Azure Cache For Redis?