The Data Mining Forum 
IMPORTANT: This is the old Data Mining forum.
I keep it online so that you can read the old messages.
Please post your new messages in the new forum: https://forum2.philippe-fournier-viger.com/index.php
How to improve the Spade algorithm for mapreduce
Posted by:
Gastel
Date: October 23, 2018 07:26AM
I want design a parallel version of SPADE algorithm for my data. My cloud is not big but I want to use Hadoop or Spark. Please give me advices.
Re: How to improve the Spade algorithm for mapreduce
Posted by:
webmasterphilfv
Date: October 26, 2018 06:15AM
Hi, Welcome to the forum
There are different ways for improving SPADE. But since you want to work with map reduce, you could consider a few ideas:
- improve load balancing between cluster nodes
- improve the algorithm or structures
- improve SPADE with new features that are not trivial.
But in any case, there are already quite a few algorithms for mining itemsets in the cloud. You could check them and compare with them.
Regards,
There are different ways for improving SPADE. But since you want to work with map reduce, you could consider a few ideas:
- improve load balancing between cluster nodes
- improve the algorithm or structures
- improve SPADE with new features that are not trivial.
But in any case, there are already quite a few algorithms for mining itemsets in the cloud. You could check them and compare with them.
Regards,
Re: How to improve the Spade algorithm for mapreduce
Posted by:
Rashid
Date: November 07, 2018 05:01PM
Can I also use map reduce for graph mining?
Re: How to improve the Spade algorithm for mapreduce
Posted by:
webmasterphilfv
Date: November 08, 2018 06:41AM
Yes, most likely. The challenge is to develop a parallel version of an algorithm and adapt it for the MapReduce or other big data framework like Spark. You could start from a non parallel algorithm and try to transform it in a parallel algorithm. But not algorithm are easy to transform in a parallel algorithm. Or you could design something new. But there are already some algorithms for big data. So instead of reinventing the wheel you should first check the existing big data algorithms for graph mining.