![]() ![]() Spark is also versatile enough to work with filesystems other than Hadoop, such as Amazon S3 or Databricks (DBFS).īut the idea is always the same. This presents new concepts like nodes, lazy evaluation, and the transformation-action (or "map and reduce") paradigm of programming. Remember, Spark is not a new programming language you have to learn it is a framework working on top of HDFS. You could also run one on Amazon EC2 if you want more storage and memory. However, if you are proficient in Python/Jupyter and machine learning tasks, it makes perfect sense to start by spinning up a single cluster on your local machine. These options cost money-even to start learning (for example, Amazon EMR is not included in the one-year Free Tier program, unlike EC2 or S3 instances).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |