Python College

Is it possible to save files in Hadoop without saving them in local file system? I would like to do… Read more Is It Possible To Save Files In Hadoop Without Saving Them In Local File System?

I would like to populate a database of Postgresql by using a mapper with MrJob and Hadoop 2.7.1. I … Read more How To Populate A Postgresql Database With Mrjob And Hadoop

We have a hive warehouse, and wanted to use spark for various tasks (mainly classification). At tim… Read more Reading And Writing From Hive Tables With Spark After Aggregation

Below is my spark data frame a b c 1 3 4 2 0 0 4 1 0 2 2 0 My output should be as below a b c 1 3 … Read more Pyspark Dynamic Column Computation

I have a quick Hadoop Streaming question. If I'm using Python streaming and I have Python packa… Read more Managing Dependencies With Hadoop Streaming?

I am very frustrated by Spark. An evening wasted thinking that I was doing something wrong but I ha… Read more Spark On Windows 10. 'files\spark\bin\..\jars""\' Is Not Recognized As An Internal Or External Command

I wrote a python code to read and write to a hadoop file system with IP hdfs_ip. It takes 3 argumen… Read more Why Am I Getting These Strange Connection Errors When Reading Or Writing To Hadoop File System With A Python Script?

I'm trying to do a Spark tutorial that comes with the Cloudera Virtual Machine. But even though… Read more Exceptions When Reading Tutorial Csv File In The Cloudera Vm

I'm on a W8 machine, where I use Python (Anaconda distribution) to connect to Impala in our Had… Read more Connect To Impala Using Impyla Client With Kerberos Auth

I am trying to do what should be a simple task: I need to convert a text file to upper case using H… Read more Hadoop Streaming With Python: Keeping Track Of Line Numbers

In my MapReduce jobs, I pass a product name to the Mapper as a string argument. The Mapper.py scrip… Read more Mapreduce How To Allow Mapper To Read An Xml File For Lookup

After reading through the documentation I do not understand how does Spark running on YARN account … Read more How Does Spark Running On Yarn Account For Python Memory Usage?

I posted a similar question about an hour ago, but have since deleted it after realising I was aski… Read more Loading A Defaultdict In Hadoop Using Pickle And Sys.stdin

I have installed, pip install thrift pip install PyHive pip install thrift-sasl and since pip ins… Read more Pyhive, Sqlalchemy Can Not Connect To Hadoop Sandbox

I am writing MapReduce job in Python, and want to use some third libraries like chardet. I konw tha… Read more Hadoop: How To Include Third Party Library In Python Mapreduce

Hey I'm fairly new to the world of Big Data. I came across this tutorial on http://musicmachin… Read more Running A Job Using Hadoop Streaming And Mrjob: Pipemapred.waitoutputthreads(): Subprocess Failed With Code 1

My map task completes successfully and I can see the application logs, but reducer stays in pending… Read more Hadoop-streaming : Reduce Task In Pending State Says "no Room For Reduce Task."

My question is similar to : hadoop streaming: how to see application logs? (The link in the answer … Read more Hadoop Streaming: Where Are Application Logs?

I have a Mapper that is going through lots of data and emitting ID numbers as keys with the value o… Read more How To Get The Reducer To Emit Only Duplicates

I have worked on Hadoop MR for quite some time and I have created and used custom(extension) Writab… Read more Create Custom Writable Key/value Type In Python For Hadoop Map Reduce?